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PREFACE 


This  report  has  been  prepared  by  Wilson  Hill  Associates, 
Inc.  for  the  Transportation  Systems  Center's  (TSC)  Service 
Assessment  Division  in  support  of  the  Impact  Assessment  Program 
conducted  by  the  Urban  Mass  Transportation  Administration 
(UMTA)  Office  of  Technical  Assistance,  Analysis  Division. 

UMTA  is  cooperating  with  and  supporting  the  transit  industry's 
efforts  to  improve  the  productivity  and  quality  of  telephone/ 
information  marketing  services  to  the  public.  The  principal 
focus  of  this  support  is  on  the  implementation  of  Automated 
Transit  Information  Systems  (ATIS)  technology.  TSC,  in  support 
of  UMTA ' s program,  is  evaluating  ATIS  demonstrations  in  Los 
Angeles  and  Washington  DC.  This  report  is  an  interim  product 
in  an  ongoing  ATIS  impact  assessment  program. 

While  specifically  intended  for  the  information  of  TSC 
and  the  management  staff  of  the  Southern  California  Rapid 
Transit  District,  this  report  is  also  generally  directed  at 
the  transit  public  information/marketing  community  and  others 
interested  in  the  application  of  computer-aided  information 
retrieval  systems. 

The  evaluations  described  in  this  report  were  conducted 
under  the  management  of  Mr.  Robert  Furniss  and  Mr.  Robert 
Phillips,  Project  Managers  for  Wilson  Hill  Associates.  Mr. 
Furniss  was  Project  Manager  through  February,  1981  when  he 
resigned  his  position  at  Wilson  Hill  to  accept  employment 
elsewhere.  On  this  date,  Mr.  Phillips  assumed  project  manage- 
ment responsibilities.  Subsequently,  Mr.  Furniss  was  hired 
through  a subcontracting  agreement  with  Wilson  Hill  to  provide 
additional  services  relating  to  the  project,  namely  post- 
implementation interviews  of  SCRTD  management  staff. 
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Much  gratitude  is  expressed  by  the  authors  to  those  SCRTD 
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Ms.  Paddie  Brennen,  Mr.  Doug  Anderson,  Mr.  Bob  Williams,  Ms. 
Barbara  Hagen,  and  Ms.  Rose  Marie  Candejas.  Thanks  also  go  to 
Ms.  Suzanne  Diller  and  other  employees  of  the  system  design 
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EXECUTIVE  SUMMARY 


This  report  is  an  interim  product  in  an  ongoing  program  to 
assess  the  impacts  of  Automated  Transit  Information  System  (ATIS) 
technology  on  the  transit  industry's  efforts  to  improve  the  pro- 
ductivity and  quality  of  telephone  information/marketing  services 
to  the  public.  It  was  prepared  by  Wilson  Hill  Associates,  Inc.  for 
the  Transportation  Systems  Center's  (TSC)  Service  Assessment  Division 
in  support  of  the  Urban  Mass  Transportation  Administration's  (UMTA) 
Impact  Assessment  Program.  The  TSC,  as  part  of  this  UMTA  program, 
is  evaluating  ATIS  demonstrations  currently  underway  in  Los  Angeles 
and  Washington,  DC. 

This  report  documents  a number  of  data  collection  and  analysis 
efforts  undertaken  in  the  evaluation  of  the  ATIS  demonstration  in 
Los  Angeles  at  the  Southern  California  Rapid  Transit  District.  This 
ATIS,  known  as  the  Computerized  Customer  Information  System  (CCIS) , 
is  currently  undergoing  testing  in  a pilot  implementation  covering 
the  San  Fernando  Valley,  a portion  of  the  SCRTD  service  area. 

While  specifically  intended  for  the  information  of  UMTA/TSC  and  the 
management  staff  of  the  Southern  California  Rapid  Transit  District, 
this  report  is  also  generally  directed  at  the  transit  public  infor- 
mation/marketing community  and  others  interested  in  the  application 
of  computer-aided  information  retrieval  systems. 

A telephone  information  service  is  one  tool  frequently  employed 
by  large  transit  authorities  as  part  of  their  overall  public 
marketing  strategy.  Telephone  information  has  several  distinct 
advantages  as  compared  with  printed  advertising  media: 

• It  is  readily  accessible  to  a substantial  fraction  of  the 
transit-riding  public; 

• It  can  be  frequently  updated  to  reflect  changes  in  schedules, 
routes,  etc. ; 

• It  can  be  personalized  to  meet  the  demands  of  the  individual 
caller . 
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The  Telephone  Information  Section  of  the  Customer  Relations 
Department  at  the  SCRTD  is  one  of  the  largest  telephone  transit 
marketing  centers  in  the  nation.  Its  85  agents  answer  caller 
queries  about  transit  service  in  3 shifts,  24  hours  a day,  365  days 
a year.  On  an  average  day,  between  10,000  and  13,000  calls  are 
received  by  the  Section.  During  peak  demand  periods,  agents  answer 
over  800  calls  per  hour.  Despite  this  effort  over  30%  of  all 
incoming  calls  are  "lost"  in  peak  periods,  that  is  callers  put  on 
hold  by  telephone  answering  equipment  hang  up  before  ever  reaching 
an  agent  or  receiving  any  information. 

Telephone  information  agents  at  the  SCRTD  have  traditionally 
retrieved  information  to  answer  caller  queries  by  referencing  large 
indexed  volumes  of  schedules,  headway  sheets,  route  descriptions, 
etc.  located  at  their  work  stations.  Advances  in  computer  tech- 
nology over  the  past  decade  have  enhanced  the  potential  for 
automating  these  data  retrieval  functions.  Certain  aspects  of 
these  functions  lend  themselves  well  to  computer  applications: 

• they  are  highly  repetitive; 

• they  require  a large  amount  of  memorization  of  routes, 
schedules,  geographic  characteristics,  etc.; 

• they  often  involve  optimization  of  a certain  variable  (for 
example,  travel  time) ; 

• they  should  demonstrate  a high  degree  of  consistency;  and 

• they  should  be  accomplished  as  quickly  as  possible. 

Application  of  computer  logic,  memory,  and  speed  to  these 

functions  thus  raises  the  potential  for  improving  the  agent's  job 
in  a variety  of  ways.  These  improvements  could  also  produce 
benefits  not  only  for  the  agents,  but  also  for  the  SCRTD  management 
and  the  general  public.  Among  these  potential  benefits  are: 

• Improved  productivity  in  terms  of  agent  calls  per  hour 
answered  due  to  faster  data  retrieval; 

• Increased  reliability  and  consistency  of  agent  responses; 

• A reduction  in  training  time  for  new  agents; 


xiii 


• Improved  capability  to  rapidly  incorporate  changes  in  transit 
agent  reference  data; 

• Expanded  capability  to  gather  statistics  concerning  individual 
agent  and  Information  Section  performance; 

• Use  of  the  transit  system  data  base  by  other  SCRTD  departments 
as  a planning  resource. 

Computer-assisted  data  retrieval  changes  the  information  agent's 
job  in  a variety  of  ways.  ATIS  operation  involves  an  agent  retrieving 
desired  information  through  use  of  a remote  interactive  computer 
terminal  located  at  the  agent's  work  station.  The  system  is 
activated  by  the  agent  entering  caller  query  data  on  the  keyboard 
and  selecting  appropriate  functions  to  produce  the  desired  type  of 
response.  Following  processing,  which  takes  a matter  of  seconds, 
the  computer  outputs  several  appropriate  responses.  The  agent  then 
evaluates  these  responses  in  light  of  the  query  and  selects  the 
most  appropriate  one  to  provide  to  the  caller. 

The  primary  purposes  of  this  evaluation  were  twofold: 

First,  to  determine  the  effects  of  these  job  changes  on  agent 
performance  and  job  satisfaction;  and  second,  to  measure  the  extent 
to  which  the  pilot  San  Fernando  Valley  area  demonstration  had  pro- 
duced the  intended  benefits  of  system  implementation,  stated  above. 

In  this  effort,  a large  amount  of  quantitative  and  qualitative 
data  were  collected.  Quantitative  data  collected  included  statis- 
tics on  Information  Section  performance  gathered  by  the  Automatic 
Call  Distribution  (ACD)  system  and  on  CCIS  agent  performance  com- 
piled in  a daily  CCIS  log  file  over  a 90-day  steady-state  evaluation 
period.  Qualitative  data  were  gathered  through  opinion  surveys 
administered  to  the  information  agents  before  and  after  this 
evaluation  period  and  through  interviews  with  key  SCRTD  personnel 
involved  in  system  implementation.  Data  of  both  types  were  also 
gathered  in  a controlled  experiment  in  which  nine  information 
agents  representing  three  different  experience  levels  and  three 
different  modes  of  data  retrieval  (CCIS  only.  Manual  only  or  a 
combination  of  both)  were  asked  to  respond  to  a standardized  set  of 
itinerary  queries  developed  by  SCRTD  and  the  evaluation  contractor. 
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A careful  review  of  these  quantitative  and  qualitative  data 
leads  to  generally  positive  findings  concerning  the  overall  per- 
formance of  the  CCIS  in  its  pilot  demonstration.  While  the  im- 
plementation process  was  rather  extended  and  somewhat  laborious  due 
to  a variety  of  internal  and  external  problems,  trends  in  the  data 
collected  show  that  most  of  those  problems  have  been  corrected  and 
the  system  is  for  the  most  part,  currently  functioning  as  originally 
intended.  System  performance  in  achieving  a number  of  different 
implementation  objectives  is  discussed  below; 

Agent  Productivity 

Data  collected  during  the  course  of  the  evaluation  leads  to 
rather  conflicting  conclusions  concerning  system  influences  on 
agent  productivity.  Under  controlled  experimental  conditions,  the 
more  experienced  CCIS  agents  were  appreciably  faster  than  were 
their  manual  and  mixed  mode  counterparts.  Similarly,  projected 
call  counts  for  CCIS  agents  on  the  CCIS  log  file  averaged  almost  30 
per  hour,  a 50%  increase  over  the  SCRTD ' s manual  standard  of  20. 

Yet  statistics  collected  for  CCIS  agents  during  the  90-day  evaluation 
period  show  that  they  did  not  perform  at  this  30-per-hour  figure, 
in  which  time  spent  between  calls  is  rather  unrealistically  assumed 
to  be  zero;  rather,  their  call  counts  were  closer  to  the  20  per 
hour  standard,  largely  a result  of  extended  agent  breaks  between 
some  calls.  The  CCIS  agents'  overall  percentage  of  calls  answered 
vs.  calls  lost  is  quite  comparable  to  that  of  the  manual  agents; 
both  groups  answered  an  average  of  about  60%  of  total  incoming 
calls.  In  addition,  their  self-admitted  call  counts  on  both  the 
agent  "before"  and  "after"  opinion  surveys  averaged  approximately 
20  per  hour. 

One  conclusion  to  be  drawn  from  these  conflicting  data,  de- 
veloped through  consultation  with  the  CCIS  Program  Manager,  is  that 
the  system  can  indeed  enable  experienced  agents  to  approach  the 
higher  rate  of  30  calls  per  hour.  However,  the  mental  stress 
experienced  by  an  agent  working  at  this  pace  is  too  great  for  it  to 
be  sustained  for  extended  periods.  Nevertheless,  a prime  advantage 
of  the  CCIS  over  manual  retrieval  is  that  all  calls,  even  those 
involving  the  most  complex  itineraries,  are  of  relatively  uniform 
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length.  CCIS  agents  are  thus  enabled,  much  more  so  than  their 
manual  counterparts,  to  develop  a steady  working  pace.  In  the  long 
run,  the  Program  Manager  stated,  agent  development  of  typing  and 
call  management  skills  will  produce  gradual  increases  in  this 
steady  pace.  The"  implication  is  that  the  CCIS  raises  the  potential, 
which  as  yet  has  not  been  achieved,  for  increases  in  overall 
Information  Section  productivity. 

Response  Accuracy  and  Consistency 

The  CCIS  data  base  is  extremely  large  and  complex,  and  problems 
encountered  in  its  creation  (digitizing  street  addresses,  for  ex- 
ample) left  it  filled  with  numerous  errors.  Similarly,  software 
routines  developed  for  itinerary  selection  initially  made  certain 
over-optimistic  assumptions  concerning  appropriate  walking  distances, 
transfer  times,  etc.  Over  an  extended  period,  refinements  and 
corrections  developed  in  consultation  with  Information  Section 
management  and  in  response  to  agent  "trouble  reports"  have  gra- 
dually reduced  the  accuracy  problem  to  nearly  zero. 

Agent  responses  to  the  "before"  and  "after"  opinion  surveys 
concerning  system  response  accuracy  show  a decided  shift  towards 
acceptance  of  CCIS  responses.  In  the  "before"  survey,  only  40%  of 
the  agents  felt  CCIS  accuracy  was  better  than  manual,  while  60% 
felt  that  it  was  about  the  same  or  worse.  In  the  "after"  survey, 

77%  of  the  agents  felt  that  CCIS  accuracy  was  better,  while  only 
23%  felt  that  it  was  about  the  same  or  worse. 

It  cannot  be  overemphasized  that  the  debugging  and  enhancement 
process  was  an  interactive  one  in  which  agents  and  Information 
Section  management  were  active  participants.  This  interaction  not 
only  resulted  in  greater  accuracy  of  the  data  base  and  a more 
workable  system,  but  also  led  to  increased  confidence  on  the  part 
of  the  agents  that  the  system  was  being  designed  to  meet  their 
individual  needs,  and  that  they  were  contributing  significantly  to 
its  accuracy.  It  is  highly  doubtful  that  this  pilot  demonstration 
would  have  enjoyed  nearly  the  success  that  it  has  without  the 
establishment  of  workable  mechanisms  to  facilitate  this  interaction. 
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The  CCIS  Working  Committee,  Program  Manager,  and  the  "trouble 
report"  system  are  all  examples  of  such  mechanisms.  It  is  signi- 
ficant that  the  system  designers,  in  retrospect,  felt  that  they 
should  have  enrolled  in  the  agent  training  course. 

Agent  Training 

Interviews  with  Information  Section  management  indicated  that 
perhaps  the  greatest  positive  impacts  exerted  by  the  CCIS  on  the 
Section's  operations  would  be  in  the  area  of  agent  training.  By 
drastically  reducing  the  need  for  agent  memorization  of  geo- 
graphical and  route  data,  the  standard  agent  training  period  can  be 
cut  at  least  in  half.  If  trainees  already  possess  typing  skills, 
this  period  may  be  even  shorter;  the  CCIS  Program  Manager  cited  the 
experience  of  training  two  Kelly  Girl  temporary  employees  to 
"reasonable"  proficiency  on  the  system  in  only  one  week. 

This  reduced  training  time  has  strong  potential  for  benefits 
in  the  areas  of  improved  personnel  recruitment  and  retention.  The 
work  force  from  which  agents  are  recruited  can  be  greatly  expanded 
to  include  older  and  part-time  employees,  neither  of  which  was 
suited  for  the  rigors  of  the  agent  training  course  used  before. 
Statistics  show  that  older  employees  are  generally  more  dependable, 
and  inclusion  of  part-time  employees  in  the  work  force  would  allow 
management  more  flexibility  than  it  presently  enjoys  in  the 
scheduling  of  full-time  agent  vacations  and  other  benefits.  The 
long-term  implication  of  these  training  benefits  is  significant 
cost  savings. 

Updating  of  Data  Base 

Another  potential  major  improvement  in  Information  Section 
operations  brought  about  as  a result  of  CCIS  implementation  is 
the  capability  to  perform  reference  data  base  updating  functions 
on  a centralized  basis.  One  full-time  employee  at  a single 
terminal  can  perform  the  data  base  updating  tasks  presently 
performed  almost  full-time  by  15  supervisors. 

A side  benefit  of  centralized  data  base  updating  mentioned 
by  almost  all  of  the  Information  Section  management  in  their 
interviews  would  be  an  improvement  in  the  job  of  supervisor.  At 
present,  supervisors  spend  practically  all  their  time  performing 
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update  functions.  As  a result,  their  skill  as  agents  tend  to 
gradually  diminish  to  the  point  where  they  are  below  those  of 
less  experienced  agents.  Centralized  updating  would  free  the 
supervisors  to  perform  the  functions  for  which  their  jobs  were 
originally  created-:  troubleshooting  and  solving  system  problems 
on  the  floor  as  they  occur. 

Agent  Job  Satisfaction 

CCIS  agent  responses  to  the  "before"  and  "after"  surveys 
lend  strong  support  to  the  hypothesis  that  implementation  of  the 
CCIS  has  led  to  an  increase  in  agent  job  satisfaction.  In  the 
"before"  survey,  67%  of  the  agents  rated  their  job  satisfaction 
as  excellent  or  good,  while  33%  rated  it  as  fair  or  poor.  In 
the  "after"  survey,  74%  of  the  agents  rated  their  job  satisfaction 
excellent  or  good  and  26%  rated  it  as  fair  or  poor.  In  the 
"before"  survey,  52%  of  the  agents  described  CCIS  influence  on 
their  job  satisfaction  as  "positive",  44%  responded  "no  influence", 
and  only  4%  responded  "negative".  By  contrast,  in  the  "after" 
survey,  84%  of  the  responses  to  the  same  question  indicated 
"positive"  influence,  while  only  16%  indicated  "no  influence", 
and  there  were  no  "negative"  responses.  In  addition,  75%  of  the 
manual  agents  responding  to  the  "after"  survey  felt  that  learning 
to  use  CCIS  would  increase  their  job  satisfaction,  while  25% 
felt  that  it  would  decrease  job  satisfaction. 

The  implication  of  these  results  is  that  a sizable  per- 
centage of  agents  felt  that  CCIS  influenced  their  job  positively, 
and  that  this  percentage  showed  an  increasing  trend  over  the 
course  of  the  evaluation  period.  One  agent  summarized  the  CCIS 
on  her  job  rather  humorously;  "I  don't  have  to  work  any  more;  I 
can  just  talk  all  day." 

Data  Collection 

The  development  of  the  CCIS  log  file  as  a data  collection 
tool  during  the  course  of  the  system  evaluation  demonstrated 
clearly  the  ability  of  the  system  to  keep  accurate  records  on 
all  aspects  of  system  performance.  Far  more  detailed  data  is 
now  available  to  Information  Section  management  concerning  the 
performance  of  each  CCIS  agent  and  of  the  entire  system  than  was 
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previously  available  with  the  less  reliable  ACD  data.  Furthermore, 
automated  data  collection  would  free  the  Senior  Supervisor  from 
a great  deal  of  paperwork  involving  ACD  counts,  time  sheets, 
etc. , leaving  more  time  to  perform  supervision  work  with  the 
supervisors  and  agents  on  the  floor. 

Integration  with  Other  Departments 

Although  a CCIS  Working  Committee  has  been  established  for 
the  purpose  of  inter-departmental  coordination  of  CCIS  activities, 
the  full  benefits  of  a totally  integrated  data  base  incorporating 
stops,  zones,  routes,  and  schedules  have  yet  to  be  fully  realized 
at  the  SCRTD.  This  situation  is  the  result  of  several  factors: 

• The  CCIS  is  currently  limited  to  the  San  Fernando  Valley  in 
its  pilot  demonstration; 

• The  Stops  and  Zones  and  Scheduling  Departments  currently 
use  their  own  data  processing  systems  which  are  incompatible 
with  the  Univac  1106  supporting  the  CCIS  demonstration;  and 

• Management  of  other  SCRTD  departments  appear  at  this  point 
to  have  little  detailed  knowledge  of  the  characteristics  or 
potentials  of  the  CCIS. 

Should  the  proposal  to  expand  the  CCIS  to  cover  the  entire 
SCRTD  service  area  be  adopted  by  the  Board  of  Directors,  con- 
siderable attention  should  be  given  in  system  specifications  to 
the  creation  of  a fully  integrated  transit  data  base  which  would 
be  of  use  to  all  SCRTD  departments. 

In  conclusion,  it  is  fair  to  say  that  the  CCIS  pilot  demon- 
stration has  achieved  most  of  its  major  intended  objectives,  and 
shows  potential  for  the  achievement  of  others.  Despite  early 
problems,  it  is  currently  functioning  with  the  support  and  con- 
fidence of  its  users.  Bringing  about  this  result  required 
dedicated  hard  work  and  a spirit  of  cooperation  on  the  part  of 
the  SCRTD  Management,  the  system  designers,  and  the  information 
agents.  Without  this  work  and  cooperation,  the  system  would 
never  have  achieved  the  success  which  it  enjoys  today. 
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1.0  INTRODUCTION 


This  report  presents  descriptions  of  a variety  of  data  col- 
lection and  analysis  efforts  undertaken  in  the  evaluation  of  the 
Computerized  Customer  Information  System  (CCIS) , an  automated 
transit  information  system  (ATIS)  currently  being  implemented 
and  tested  in  a pilot  deployment  at  the  Southern  California 
Rapid  Transit  District  (SCRTD).  The  objectives  of  this  report 
are  to  quantify  and  assess  the  impacts  of  this  application  of 
ATIS  technology  on  system  users  and  management  within  the  SCRTD, 
as  well  as  on  the  general  public  in  the  Los  Angeles  area  being 
served  by  the  system,  and  to  determine  how  well  the  CCIS  met  its 
pre- implementation  objectives.  It  is  hoped  that  the  descriptions 
of  system  implementation  efforts  in  this  report  will  also  serve 
to  provide  helpful  insights  for  other  transit  properties  interested 
in  the  potential  application  of  ATIS  technology  to  their  own 
public  information/marketing  efforts. 

1.1  AUTOMATION  OF  TRANSIT  INFORMATION  SYSTEMS 

1.1.1  The  Role  of  Telephone  Information  in  Transit  Marketing 
Local  transit  properties  have  at  their  disposal  a variety 

of  media  for  marketing  their  services  to  the  general  public. 

Among  these  are  printed  schedules  and  maps,  radio,  television 
and  newspaper  advertising,  and  telephone  information  disseminated 
from  a central  office.  As  compared  with  other  media,  a telephone 
information  service  has  several  distinct  advantages.  First,  it 
is  readily  accessible  to  a substantial  fraction  of  the  transit- 
riding public.  Second,  it  is  more  reliable  and  up-to-date. 

Third,  and  most  important,  it  can  be  personalized  to  suit  the 
demands  of  the  individual  customer.  For  these  reasons,  parti- 
cularly in  urban  areas  having  complex  and  perhaps  multi-modal 
transit  systems,  a telephone  information  service  is  generally 
employed  by  local  transit  properties  to  overcome  a major  barrier 
to  transit  use:  public  apprehension  of  "getting  lost"  on  public 
transit. 
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1.1.2  Traditional  Transit  Information  Center  Characteristics 


In  a typical  transit  system  telephone  information  center, 
system  employees  (who  in  this  report  will  be  referred  to  as 
"agents")  answer  inquiries  from  prospective  transit  riders  about 
schedules,  routes,  particular  trip  itineraries,  fares,  etc. 
Traditionally,  the  agent  receives  the  caller's  inquiry,  usually 
assisting  the  caller  in  stating  it  as  exactly  as  possible.  Once 
the  inquiry  is  understood,  the  agent  consults  route  maps,  schedules, 
headway  sheets  or  other  indexed  printed  information  located  at 
the  agent's  work  station,  piecing  together  the  information  desired. 
Once  an  appropriate  response  has  been  retrieved,  it  is  provided 
to  the  caller. 

Under  this  traditional  manual  system  of  data  retrieval, 
the  agent's  command  of  interpersonal  skills  is  particularly 
important.  Of  equal  importance  under  this  system  is  the  agent's 
knowledge  of  both  transit  system  and  geographic  characteristics. 

In  order  to  be  able  to  quickly  locate  a desired  trip's  original 
and  destination  and  to  effectively  select  appropriate  transit 
routings  between  them,  the  agent  must  undergo  intensive  training 
as  well  as  on-the-job  skill  development  before  full  competency 
is  achieved.  This  is  especially  true  in  cases  where  the  transit 
system  is  complex  and/or  its  service  area  is  large. 

1.1.3  The  Potential  for  Automation  of  Agent  Data  Retrieval 
Functions 

Advances  in  computer  technology  over  the  past  decade  have 
raised  the  potential  for  automating  data  retrieval  functions 
performed  by  transit  information  agents.  Certain  aspects  of 
these  functions  lend  themselves  well  to  computer  applications. 

First,  they  are  highly  repetitive;  a very  large  percentage  of 
caller  inquiries  can  be  broken  down  into  only  four  or  five 
distinct  call  types,  each  requiring  agent  consultation  of  cer- 
tain specific  data.  Thus,  programming  the  computer  for  data 
retrieval  is  a relatively  manageable  task  of  duplicating  agent 
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behavior  elicited  by  each  of  the  distinct  call  types.  Second,  data 
retrieval  functions  require  a high  degree  of  agent  familiarity  with 
service  area  geography  and  transit  system  characteristics. 
Characteristics  such  as  these  can  be  digitized  on  a coordinate  grid 
system  and  mapped  into  the  memory  of  a computer.  Third,  agent 
functions  often  involve  optimization  of  response  data  to  conform  to 
limitations  placed  on  certain  trip  parameters  by  the  caller.  For 
example,  the  caller  might  want  to  minimize  total  travel  time,  the 
number  of  transfers  required,  walking  distances,  etc.  The  logic 
underlying  agent  response  selection  in  view  of  these  caller  stated 
limitations  can  be  translated  into  comparing/optimizing  computer 
routines  in  a very  straightforward  manner.  Fourth,  it  is  very 
important  that  agent  data  retrieval  exhibit  a high  degree  of 
consistency.  While  human  error  may  introduce  a certain  amount  of 
variability  in  agent  responses,  machine  logic  ensures  that  the  same 
inquiry  will  elicit  the  same  computer  response  every  time.  Finally, 
due  to  high  public  demand  for  their  services,  agents  must  work  as 
quickly  and  efficiently  as  possible  in  retrieving  data.  The  high- 
speed calculating  capabilities  of  a computer  are  thus  particularly 
appropriate  in  this  application.  Except  for  a relatively  small 
percentage  of  cases  in  which  agents  are  able  to  answer  inquiries 
"off  the  top  of  their  heads"  automated  data  retrieval  is  faster 
than  manual  consultation  of  reference  materials.  For  detailed 
inquiries  involving  multiple- transfer  trips  it  is  substantially  (up 
to  100%)  faster. 

Thus,  automation  of  data  retrieval  functions  would  appear  to 
be  a technically  feasible  means  of  improving  a telephone  infor- 
mation agent's  capabilities  in  terms  of  response  accuracy,  con- 
sistency, and  speed.  The  agent's  job,  however,  is  much  different 
under  an  automated  transit  information  system.  ATIS  operation 
involves  an  agent  receiving  telephone  queries  from  a caller  and 
retrieving  desired  information  through  the  use  of  a remote  inter- 
active computer  terminal  located  at  the  agent's  work  position.  The 
system  is  activated  when  query  data  is  entered  on  the  terminal 
keyboard  by  the  agent.  The  query  is  then  processed  in  the  computer 
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through  preprogrammed  routines  selected  by  the  agent  which  reference 
the  geographic  and  transit  data  bases  (addresses,  fares,  routes, 
schedules,  etc.)  and  optimize  certain  caller- stated  trip  parameters 
(travel  time,  transfers,  fare,  walking  distances,  etc.)  through 
comparison  of  pote'ntially  feasible  alternatives.  Following  pro- 
cessing, which  is  accomplished  in  a matter  of  seconds,  the  most 
appropriate  responses  considering  given  cal ler- stated  limitations 
are  displayed  on  a cathode  ray  tube  located  above  the  agent's 
keyboard.  This  display  is  then  evaluated  by  the  agent  and  the 
appropriate  information  supplied  to  the  caller. 

1.1.4  ATIS  Implementation  Benefits 

In  light  of  the  above  discussion,  it  would  appear  that  a well- 
designed  and  implemented  automation  of  a transit  information  center 
would  produce  benefits  to  both  the  transit  property  and  the  general 
public.  These  benefits  include  the  following: 

1.  Improved  information  agent  service  and  productivity  re- 
sulting from  faster  data  retrieval  by  the  computer. 

2.  Increased  reliability  and  consistency  of  agent  responses, 
since  the  computer  should  always  supply  the  same  answer  to  the 
same  questions. 

3.  A reduction  in  training  time  for  information  agents,  since 
extensive  training  in  area  geography  and  transit  system 
characteristics  is  no  longer  necessary  - this  information 
would  be  stored  in  the  computer's,  rather  than  the  agent's, 
memory. 

4.  Improved  capability  to  rapidly  incorporate  schedule  or 
other  service  changes  into  the  information  center's  reference 
data  base,  since  such  changes  would  require  only  a simple 
modification  of  the  computer's  memory  rather  than  reprinting 
or  modifying  hard  copy  reference  data  for  all  information 
agents. 

5.  Expanded  capability  to  gather  statistics  about  telephone 
information  center  operations  since  the  computer  can  be 
programmed  to  store  and  compile  all  agent-directed  functions 
and  processing  transactions. 
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2.0  ATIS  IMPLEMENTATION  AT  SCRTD 


In  this  section,  a brief  history  of  impleinentat  ion  of  the 
CCIS  at  the  Southern  California  Rapid  Transit  District  (SCRTD) 
is  presented.  Table  2-1  summarizes  this  history. 

2.1  SCRTD  TELEPHONE  INFORMATION  SYSTEM  CHARACTERISTICS 

The  SCRTD ' s Telephone  Information  Facility  embodies  a number 
of  characteristics  that  made  it  a prime  candidate  for  ATIS  imple- 
mentation and  testing.  The  Rapid  Transit  District's  system  is 
both  immense  and  complex;  the  service  area  extends  up  to  90 
miles  north  and  south  and  50  miles  east  and  west  in  the  Los 
Angeles  Metropolitan  Area.  Within  this  area,  over  200  SCRTD  bus 
routes  and  2400  buses  serve  185  different  communities  and  cities. 

On  an  average  day  the  SCRTD  Telephone  Information  Facility 
receives  between  10,000  and  13,000  calls  on  83  different  trunk 
lines  covering  the  entire  service  area.  To  meet  this  demand, 
the  SCRTD  employs  a total  of  85  telephone  information  agents 
who  answer  calls  in  3 shifts,  24  hours  a day,  365  days  a year. 
Despite  this  considerable  investment  in  manpower,  between  30  and 
40  percent  of  all  incoming  calls  are  "lost"  during  peak  demand 
periods;  that  is,  callers  who  are  put  on  hold  by  the  telephone 
answering  equipment  because  all  agents  are  busy  hang  up  before 
ever  reaching  an  agent  or  receiving  any  information.  During 
these  peak  periods,  the  agents  answer  calls  at  a rate  of  over 
800  per  hour. 

In  order  to  become  thoroughly  familiar  with  SCRTD ' s complex 
system,  an  agent  must  undergo  an  intensive  eight-week  classroom 
training  session.  At  the  end  of  this  period,  an  agent  should 
have  a demonstrated  ability  to  keep  updated  on  operations,  pro- 
cedures, and  policies  of  the  SCRTD,  as  well  as  a working  knowledge 
of  the  various  bus  routes.  In  the  past,  a minimum  of  six  months 
of  on-the-job  experience  has  been  required  before  an  agent  can 
be  considered  fully  qualified  to  handle  caller  requests  for 
information. 
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TABLE  2-1.  COMPUTERIZED  CUSTOMER  INFORMATION  SYSTEM: 
CHRONOLOGY  OF  SYSTEM  DESIGN  AND  IMPLEMENTATION 

DATE 

SYSTEM  STATUS 

1970-72 

Various  strategies  for  improving  telephone 
information  service  examined;  computerized 
response  idea  supported  by  SCRTD  management. 

1973-74 

PARIS  system  demonstration  by  System 
Development  Corporation  (SDC)  in  Santa 
Monica . 

1975 

Funding  for  hardware  and  software  procurement 
included  in  SCRTD  data  processing  capital 
grant  application. 

1976 

Decision  to  implement  a CCIS  demonstration 
limited  in  scope  to  the  San  Fernando 
Valley . 

1977 

Preparation  of  system  RFP ' s and  selection 
of  SDC  as  system  design  contractor  through 
competitive  process. 

1977-78 

Resolution  of  UMTA  funding  problems  due 
to  proprietary  nature  of  SDC  software. 

1978 

Beginning  of  technical  system  design. 

1978 

Construction  of  the  digital  geographic 
and  transit  system  data  bases. 

1979 

System  operational;  refinement  and  fine 
tuning  process  involving  users  and  design 
contractor . 

1979-80 

Development  of  additional  system  improve- 
ments and  system  acceptance  by  SCRTD. 

1980 

Evaluation  contractor,  selected  by  UMTA, 
makes  recommendations  to  increase  system 
effectiveness . 

1980 

Controlled  experiment  conducted  by  evaluation 
contractor  to  evaluate  comparative  performances 
of  CCIS  and  manual  agents. 
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1981 


1982 


1982 


Effectiveness  recommendations  implemented 
by  SCRTD,  with  positive  results  on  operator 
productivity  and  enthusiasm. 

Completion  of  system  evaluation  and 
development  of  additional  system  refinements, 
including  installation  of  new  telephone 
equipment . 

Decision  by  the  SCRTD  Board  of  Directors  to 
expand  the  CCIS  systemwide. 
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2.2  CCIS  PROJECT  ORIGIN  AND  BACKGROUND  (1970-1977) 

The  origins  of  the  CCIS  project  can  be  traced  back  to  the 
early  1970 's  when  the  management  of  the  SCRTD  was  examining 
various  strategies  for  improving  its  services.  One  such 
strategy  considered  at  the  time  was  the  utilization  of  a 
computer  to  support  the  customer  information  program.  The 
General  Manager  of  the  SCRTD  believed  in  and  publicly  supported 
this  strategy,  thereby  laying  the  initial  groundwork  for  further 
research  in  this  direction. 

The  SCRTD ' s early  optimism  in  the  area  was  countered  by  a 
study  on  the  potential  of  computer-based  transit  information 
retrieval  systems.  This  study  was  performed  by  the  National 
Bureau  of  Standards  in  Washington  DC.*  This  NBS  study  indicated 
that,  while  small  service  area  ATIS  applications  might  be 
feasible,  computer  processing  requirements  for  selecting 
appropriate  itineraries  increased  geometrically  with  the  amount 
and  complexity  of  transit  service  provided.  The  study  concluded 
there  were  no  existing  computer  systems  with  sufficient  capacity 
to  handle  large  transit  service  areas  in  a responsive  manner;  in 
other  words,  the  fastest  computer  retrieval  speeds  achievable 
for  large  computer  systems  were  not  comparable  to  those  already 
achieved  by  agents  using  manual  retrieval  methods. 

Despite  this  conclusion,  the  computer  idea  remained  alive  at 
the  SCRTD  due  to  monitoring  of  progress  in  1973  for  a local 
research  and  development  effort  conducted  by  System  Development 


*Douglas  R.  Shier  and  Judith  F.  Gilsinn,  Cost/Benefit  Analysis 
of  Automated  Transit  Information  Systems  PB268424  NBSIR  77-1253 
final  issue  June  1977. 
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Corporation  (SDC)  in  nearby  Santa  Monica.  A researcher  at  SDC 
had  focused  his  doctoral  thesis  on  attempting  to  solve  the  algor- 
ithmic problem  of  computing  transit  itineraries  for  larger 
service  areas  within  a demand- responsive  time  frame.  This 
route- f ind ing  research  was  expanded  into  a corporate  venture  to 
develop  a computer-based  information  retrieval  system  for  the 
local  Santa  Monica  Municipal  Bus  Lines,  a small  privately-held 
transit  company.  This  effort  resulted  in  a set  of  computer 
programs  known  as  PARIS*  (Passenger  Routing  Information  System). 
This  system  became  operational  in  1974  and  was  used  in  a 
demonstration  program  by  Santa  Monica  Municipal  Bus  Lines  for 
approximately  one  year. 

The  successful  operation  of  PARIS  in  Santa  Monica  was  observed 
by  senior  SCRTD  management  including  the  Director  of  Customer 
Relations,  who  had  budget  responsibility  for  the  Telephone  Infor- 
mation Section,  and  the  Director  of  Management  Information 
Systems,  who  had  budget  responsibility  for  the  SCRTD ' s computer 
area.  Observing  PARIS  in  operation  convinced  these  senior 
managers  that  it  was  worth  the  risk  to  attempt  installation  of  a 
pilot  ATIS  to  support  the  SCRTD  Telephone  Information  Section. 

A limited,  pilot  ATIS  deployment  was  considered  desirable  at 
SCRTD  for  two  reasons.  First,  such  an  effort  would  give  the 
SCRTD  the  opportunity  to  determine  whether  ATIS  technology  could 
actually  perform  in  the  context  of  a larger  service  area  and  a 
high  level  of  transit  bus  service.  Second,  a limited  test  was 
deemed  necessary  since  there  was  understandably  much  skepticism 
in  the  Telephone  Information  Section  on  the  ability  of  this 
technology  to  perform  a function  which  had  been  traditionally 


*PARIS  is  a trademark  of  SDC. 
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accomplished  only  by  trained,  experienced  agents  using  printed 
reference  material.  Funding  was  requested  by  a separate  line 
item  in  the  SCRTD ' s 1975  data  processing  capital  grant  appli- 
cation. The  line  item  covered  both  hardware  and  software  for  the 
pilot  systems.* 

The  SCRTD  service  sub-area  selected  for  testing  this  pilot 
system  was  the  300-square  mile  San  Fernando  Valley  region  north 
of  downtown  Los  Angeles.  This  area  was  chosen  for  several 
reasons.  First,  it  is  geographically  well-defined  and  of 
manageable  size  for  a pilot  application.  Second,  all  calls  from 
the  Valley  enter  the  SCRTD  Telephone  Information  Center  via  a 
single  trunk  line,  thus  allowing  for  easy  data  recording  and 
tabulation  of  calls.  Finally,  its  4000  streets  and  32 
bi-directional  bus  routes,  including  1800  stops,  are  laid  out 
largely  in  a rectangular  grid  network,  which  simplified  the  task 
of  digitizing  geographic  and  transit  service  characteristics  in 
the  computer's  memory. 

Due  to  the  innovative  nature  of  the  project  the  capital  grant 
process  was  lengthy.  Visits  were  required  by  UMTA  personnel  to 
identify  and  discuss  the  nature  of  the  project.  The  SCRTD  was 
also  required  to  write  a subsequent  justification  of  the  project 
for  UMTA  review.  A capital  grant  for  $150,000  was  finally 
approved  in  the  spring  of  1977.  The  key  project  justification 
was  based  on  the  planned  CCIS  using  the  SCRTD ' s mainframe  computer 
(Univac  1106)  on  a timeshared  basis.  This  strategy  distinguished 
the  project  from  another  ATIS  scheduled  for  UMTA  funding  at  the 
Washington,  DC  Metropolitan  Area  Transit  Authority  (WMATA) ; this 
system  was  to  be  based  on  the  use  of  dual  dedicated  mini-computers. 


* The  software  requirement  specified  "already  existing"  software 
as  opposed  to  software  development. 
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2.3  CCIS  CONTRACTOR  SELECTION  PROCESS  (1977-1978) 

During  the  extended  capital  grant  approval  process,  the  SCRTD 
prepared  an  in-house  draft  Request  for  Proposal  (RFP)  and 
devised  alternative  procedures  for  the  selection  of  a system 
contractor.  One  advantageous  by-product  of  the  delay  in  the 
project's  implementation  was  the  opportunity  provided  the  SCRTD 
to  observe  the  experiences  of  other  transit  properties,  notably 
Philadelphia,  which  had  entered  the  market  for  automated  customer 
information  systems.  Three  very  valuable  lessons  were  learned 
during  this  period: 

• Hire  only  one  system  contractor. 

« Recognize  the  sensitivities  of  the  intended  users  (i.e., 
the  agents  and  their  supervisors)  of  this  technology  and 
strive  to  achieve  an  understanding  of  user  needs  through 
communications  with  them. 

• Accept  bids  only  from  contractors  who  could  demonstrate 
"working  systems". 

The  strategy  of  hiring  only  one  system  contractor  was  based  on 
Philadelphia's  experience  in  1975-1976  in  which  an  attempt  was 
made  to  install  an  ATIS  to  assist  transit  information  agents 
during  the  Bicentennial  period  when  a large  number  of  tourists 
were  expected  to  visit  the  city.  This  project  was  organized 
around  three  separate  contractors:  a mini-computer  firm  for  the 
itinerary  software,  a microfiche  firm,  and  a management  firm  to 
coordinate  the  other  project  consultants.  This  project  organi- 
zation proved  to  be  ineffective  in  producing  a well  integrated 
team  effort.  The  SCRTD  observed  this  problem  and  made  a decision 
to  specify  a single  system  contractor  in  the  RFP. 

The  need  to  recognize  the  sensitivity  of  the  transit  infor- 
mation agent  to  ATIS  technology  was  also  confirmed  by  this 
Philadelphia  experience.  Agents  in  Philadelphia  resisted  the 
development  and  implementation  of  the  customer  information 
system  because  the  project  was  partially  justified  on  the  basis 
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that  they  couldn't  possibly  handle  the  call  demand  expected 
during  the  Bicentennial  visitor  period.  This  justification, 
although  well  intentioned,  offended  their  pride  and  lost  their 
cooperation.  Through  observing  this  experience,  the  SCRTD 
became  more  aware  of  agent  sensitivities  to  ATIS  technology  and 
other  innovations: 

• Agents  take  great  pride  in  their  expertise  and  their 
detailed  knowledge  of  the  service  area  and  transit  service. 

• Agents  place  great  value  on  learning  through  experience. 

• Agents  can  respond  to  computer  technology  in  a very  negative 
manner  if  it  is  perceived  as  a threat  to  either  job  security 
or  seniority. 

The  Supervising  System  Analyst  who  authored  the  SCRTD ' s RFP 
attempted  to  address  these  sensitivities  by  involving  intended 
users  in  the  process  of  identifying  informational  requirements 
which  had  to  be  satisfied  by  any  form  of  customer  information 
system.  Draft  portions  of  the  RFP  were  circulated  to  the 
Telephone  Information  Section  for  review  and  comment. 

This  process  was  not  highly  productive  in  terms  of  receiving 
input  on  informational  requirements,  but  it  was  significant  in 
that  the  users  were  involved  in  the  procurement  effort  from  the 
outset.  This  sensitivity  to  the  users  was  also  reflected  in 
the  membership  of  the  Contractor  Selection  Committee,  described 
below. 

The  decision  to  accept  bids  only  from  contractors  who 
could  demonstrate  "working  systems"  was  also  the  result  of 
monitoring  other  ATIS  procurement  efforts.  The  bidding  on  the 
Philadelphia  system  and  the  bidding  interest  in  the  planned 
WMATA  system  (called  AIDS)  demonstrated  there  was  several  firms 
who  were  marketing  ATIS  packages  which  had  already  been  developed 
and  tested,  though  not  necessarily  sold  to  a transit  agency. 

The  SCRTD  wished  to  avoid  getting  involved  in  a research  and 
development  effort  with  an  unproved  system's  contractor  and 
therefore  specified  the  "working  system"  requirement  for  bidders. 
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In  addition  to  the  three  RFP  provisions  which  resulted 
from  observations  of  the  Philadelphia  experience,  two  other 
requirements  of  the  SCRTD ' s RFP  were  significant: 

• The  system  hardware  and  software  had  to  be  compatible 
with  the  requirements  of  the  SCRTD ' s in-house  main- 
frame (Univac  1106)  computer.  This  provision  was  in 
deference  to  the  capital  grant  approval  justification 
which  specified  an  ATIS  mainframe  test.  The  con- 
tractor could,  however,  recommend  the  type  of 
terminals  to  be  used. 

• The  response  time  standard  for  retrieval  of  informa- 
tion on  the  ATIS  was  established  at  seven  seconds  or 
less  assuming  a "dry"  system  (no  other  jobs  being  run 
on  the  mainframe  computer) . 

The  RFP  was  completed  and  distributed  in  June  1977.  The  response 
period  was  initially  eight  weeks,  but  was  later  extended  to 
twelve  weeks  at  the  request  of  one  bidder. 

During  the  RFP  preparation  process,  the  SCRTD  formed  a 
Selection  Committee  to  evaluate  contractor  proposals  and  system 
demonstrations,  and  make  a recommendation  for  contractor  selec- 
tion. The  Committee  consisted  of  the  following  SCRTD  personnel 
(organized  by  department) : 

• Management  Information  Systems 

Supervising  Systems  Analyst 


• Telephone  Information 

Chief  of  Telephone  Information 
Senior  Instructor 
Senior  Supervisor 

• Planning  - Scheduling  Departments 

Advance  Planner 

• Purchasing  Department 

Purchasing  Representative 

The  committee  membership  reflected  a continuation  of  the  effort, 
mentioned  above,  to  involve  the  user  in  the  evaluation  of  con- 
tractors and  proposed  systems. 
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A total  of  five  firms  submitted  proposals  to  the  SCRTD  for 
review  by  the  September  16,  1977  proposal  deadline.  The  general 
criteria  established  by  the  Committee  to  evaluate  contractor 
proposals  included  the  following  (not  necessarily  in  order  of 
priority)  : 

• Workability  of  proposed  system: 

Was  the  system  technically  sound? 

Were  the  algorithms  used  to  computer  itineraries 
workable? 

Was  system  operation  based  on  proven  logic? 

• Computer  Equipment: 

Use  of  the  mainframe  computer  and  level  of  Univac 
system  expertise 

Recommended  terminals  for  system  and  rationale  for 
this  recommendation 

• Training  Requirements 

Ability  to  design  and  implement  training  program 

for  agents  at  their  level  of  comprehension  and 
skill 

Ability  to  work  with  agents 
Interface  with  SCRTD  Training 

• Level  of  Knowledge  on  Transit  and  Telephone  Information 

Operations: 

Knowledge  of  transit  bus  operations 

Knowledge  of  transit  data  base  components  (schedules, 
routes,  stops  and  zones,  fares) 

Knowledge  of  transit  telephone  information  operations 

Knowledge  of  transit  service  area  and  geographic 
data  base 

• Ability  to  perform  within  Project  Budget  ($150,000) 

• Ability  to  Support  System  after  Implementation  Logistics 
of  Support 

The  fact  that  five  firms  responded  to  the  RFP  enabled  the  SCRTD 
to  take  a comprehensive  look  at  the  state  of  the  art  in  ATIS 
technology.  Each  firm  had  at  least  one  particular  strength; 
the  Committee  eventually  concluded  they  learned  valuable  infor- 
mation from  dealing  with  each  bidder.  Without  detailing  the 
specific  ratings  of  each  firm,  it  is  useful  to  summarize  the 
major  weaknesses  identified  in  the  process  of  evaluating  all 
the  losing  bidders: 
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• Deficiencies  in  meeting  training  requirements  (This 
weakness  applied  to  all  but  one  of  the  losers) 

• Unworkable  algorithm  for  system 

• Inabil'ity  to  interface  effectively  with  the  users 

• Inadequate  knowledge  of  transit  systems 

• Resistance  to  use  of  Univac  mainframe  and  inflexible 
approach  on  specifications  of  terminals  for  the  system 

• Lack  of  confidence  in  presenting  the  system 

• Inadequate  provisions  for  continued  system  support 
When  the  rating  process  was  completed,  the  Committee  made  a 
recommendation  to  the  SCRTD  Purchasing  Department  to  select 
System  Development  Corporation  (SDC),  the  same  company  which 
had  run  the  Santa  Monica  demonstration.  This  recommendation 
was  based  on  the  following  strengths  in  the  sDC  proposal: 

• Ability  to  deliver  a workable  system  which  was  judged 
as  technically  sound. 

• A combined  expertise  of  hardwar e/sof twar e and  transit 
knowledge  of  SCRTD  operations. 

• Status  as  local  California  firm  provided  cost  advan- 
tages in  terms  of  system  implementation  and  ongoing 
suppor  t . 

• An  effective  plan  for  the  use  of  project  resources 
permitting  a focus  on  the  labor  intensive  data  base 
tasks. 

• A confident,  well-organized  system  presentation  to 
the  Selection  Committee. 

A system  presentation  was  also  conducted  for  the  SCRTD  Board  of 
Directors  who  subsequently  agreed  to  the  recommendation  of  the 
Selection  Committee. 

The  contract  between  SCRTD  and  SDC  was  prepared  and  sent 
to  Washington,  DC  in  October  1977  for  approval  by  UMTA.  At 
this  point,  two  legal  problems  developed  which  delayed  the 
award  of  the  contract  until  May  1978.  The  first  of  these  problems 
involved  the  fact  that  the  RFP  had  not  been  approved  by  UMTA 
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prior  to  being  distributed.  The  second  problem  was  that  the 
legal  nature  of  the  contract  was  complicated  by  the  fact  that 
the  SCRTD  was,  in  essence,  purchasing  a 1 icen se  to  operate  a 
proprietary  product  (i.e.,  the  SDC ' s PARIS  program),  a process 
not  usually  undertaken  by  the  Federal  Government,  which  is 
generally  more  interested  in  funding  demonstration  projects 
with  the  potential  for  application  to  other  Federally- funded 
transit  systems.  After  extended  deliberations,  UMTA  proposed  a 
way  of  resolving  these  two  RFP  approval  problems.  The  UMTA 
proposal  involved  the  SCRTD  cancelling  the  original  RFP,  rewriting 
a second  RFP,  and  re- advertising  for  bids  with  the  intention  of 
completing  a two-phased  procurement.  The  first  phase  of  the 
procurement  would  cover  the  pilot  project  as  previously 
advertised  (the  San  Fernando  Valley  system),  while  the  second 
phase  would  cover  the  SCRTD ' s entire  service  area.  The  SCRTD 
considered  this  approach  and  opposed  it  on  two  grounds.  First, 
it  was  felt  that  the  pilot  project  was  required  simply  to 
determine  whether  an  ATIS  could  be  made  to  work  to  the  satis- 
faction of  SCRTD  users  and  management,  and  making  a full 
commitment  to  the  entire  service  area  was  considered  premature. 
Second,  any  commitment  to  lock  the  full  service  area  into  the 
1977-78  level  of  technology  was  questionable,  considering  the 
pace  at  which  cost-saving  developments  are  made  in  the  computer 
field.  Such  developments  might  well  render  the  procured  system 
obsolete  in  only  a few  years.  The  second  legal  problem,  regarding 
the  purchase  of  a license,  meant  that  the  SCRTD  would  not  be 
purchasing  the  system  programs  themselves,  but  only  the  right 
to  operate  them  for  the  San  Fernando  Valley  portion  of  the 
service  area.  Additional  time  was  required  to  enable  SCRTD  to 
comply  with  the  federal  requirements  on  purchasing  such  a 
license.  The  dispute  over  the  RFP  was  finally  resolved  on 
February  20,  1978  when  UMTA  abandoned  its  previous  objections 
and  approved  the  original  RFP.  Upon  resolution  of  the  legal 
issues,  it  was  necessary  to  recertify  the  bids  on  the  project. 

When  this  task  v.'as  completed  the  system  contract  was  oficially 
approved  and  funding  awarded  in  May  1978. 
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2.4  INITIAL  TECHNICAL  APPROACH  AND  COORDINATION 

The  contractor  team  was  assembled  in  May  1978  to  begin  the 
project;  the  team  consisted  of  four  systems  personnel  with  their 
combined  expertise  covering  the  areas  of  hardware,  software, 
data  base  creation  requirements,  the  PARIS  programs,  and  the 
system  training.  The  initial  interface  between  the  contractor 
and  the  SCRTD  was  necessarily  a technical  one  involving  the 
SCRTD ' s data  processing  personnel.  The  overall  technical  task 
facing  this  group  was  that  of  entering  the  contractor's  system 
of  programs  on  the  client's  computer.  This  task  was  complicated 
by  the  fact  that  the  original  PARIS  programs  were  designed  for 
IBM  equipment.  The  SCRTD  application  was  based  on  the  use  of  a 
Univac  mainframe  thus  requiring  the  conversion  of  all  programs 
to  a format  compatible  with  the  Univac  compiler.  Technical 
coordination  meetings  on  systems- related  issues  lasted  approxi- 
mately eight  weeks  through  June  and  July  of  1978.  The  process 
eventually  became  so  involved  that  the  team  finally  agreed  to  a 
simple  technical  approach:  the  system  would  initially  be 

installed  in  accordance  with  the  contractor's  proposal  and  later 
adjusted  to  any  implementation  problems  as  they  appeared.  The 
decision  was  made  to  rewrite  the  PARIS  programs  into  Univac 
ASCII  Fortran  and  110  series  assembly  language.  The  contractor 
responded  to  this  requirement  by  staffing  the  project  with  a 
systems  analyst  having  extensive  Univac- r el  a ted  software  experience. 

This  initial  project  focus  on  technical  issues  produced  a 
major  problem  early  in  the  implementation  process.  Systems 
personnel  had,  in  effect,  stopped  talking  to  users  (the  Telephone 
Information  personnel)  during  this  two-month  technical  design 
period.  This  silence  made  the  users  feel  unrecognized  as  project 
participants  and  unappreciated  in  terms  of  operational  input 
they  could  provide.  This  exclusion  from  the  project  design 
contributed  to  a lack  of  user  confidence  in  the  potential  of  the 
CCIS  to  support  telephone  information  operations. 
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This  inadvertent  user  exclusion  was  recognized  by  system 
personnel  in  August  1978.  To  address  the  problem  a decision  was 
made  to  hold  weekly  project  meetings  with  Telephone  Information 
personnel.  These  meetings  made  systems  personnel  realize  they 
had  a responsibility  to  try  to  translate  computer  terminology 
and  technical  discussions  into  layman's  language;  they  also 
realized  the  need  to  learn  more  about  the  user's  world  before 
designing  system  features  of  the  CCIS. 

These  weekly  meetings  also  made  the  senior  Telephone  Infor- 
mation section  staff  realize  they  had  a reciprocal  responsibility 
to  learn  the  basics  of  computer  systems  and  data  processing; 
this  knowledge  would  enable  them  to  understand  and  effectively 
contribute  to  the  technical  design  process.  In  the  fall  of  1978 
the  senior  management  and  instructional  staff  of  the  Telephone 
Information  section  enrolled  in  an  introductory  data  processing 
course  at  a local  university;  each  staff  member  successfully 
completed  the  course.  At  the  same  time  the  systems  staff  were 
devoting  time  to  learning  more  about  data  processing  requirements 
from  the  transit  information  agents.  It  might  be  noted  that 
systems  personnel  felt,  in  retrospect,  that  the  best  way  for 
them  to  become  familiar  with  user  requirements  would  have  been 
to  enroll,  like  new  agents,  in  the  official  SCRTD  eight-week 
training  course. 

Within  three  months  the  implementation  approach  was  thus 
transformed  into  a more  participatory  effort  involving  effective 
dialog  between  system  technicians  and  users.  The  time  required 
in  this  necessary  coordination  effort  necessitated  a twelve  week 
extension  in  the  overall  project  schedule. 

2.4.1  Establishment  Of  The  Geographic  Data  Base 

The  next  step  in  the  system  implementation  process,  creation 
of  a geographic  data  base  for  the  San  Fernando  Valley,  involved 
the  laborious  task  of  digitizing  all  the  street  addresses  in  the 
service  area.  Street  address  ranges  were  encoded  on  a coordinate 
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grid  for  the  straight  segment  of  every  street,  along  with  start 
and  end  addresses  for  each  segment.  This  process  is  aided  by  the 
use  of  a machine  called  a "digitizer"  which  automatically 
identifies  x-y  coordinates  when  its  pointer  is  placed  on  a map. 

At  the  time  this  task  was  initiated  (the  summer  of  1978) 
digitized  geographic  information  was  not  available  from  the  U.S. 
Census  Bureau.  The  contractor  was  thus  faced  with  the  options  of 
digitizing  either  manually  or  by  machine  the  street  address 
network  and  various  local  landmarks.  Efforts  to  locate  an 
appropriate  digitizing  machine  led  the  contractor  to  the 
University  of  California  at  Los  Angeles  (UCLA) , where  a flat  bed 
digitizing  machine  system  was  available.  An  agreement  was  secured 
by  the  contractor  permitting  use  of  this  system  two  hours  a day 
for  two  months. 

The  contractor  operated  the  digitizing  machine  using  large 
wall  size  commercial  maps  of  the  San  Fernando  Valley;  these 
commercial  maps  were  considered  to  be  the  best  available  data 
base  on  the  current  street  network  in  the  Valley.  These  maps 
were  larger  versions  of  those  contained  in  the  commercial  map 
guide  (Thomas  Guide)  used  by  the  transit  information  agents. 

The  two  month  machine  digitizing  effort  originally  anticipated 
was  not  sufficient  to  complete  the  task  due  to  the  size  and 
extent  of  the  street  address  network  and  the  limited  access  time 
to  the  machine.  In  addition,  the  productivity  of  the  machine 
digitizing  effort  was  constrained  by  the  use  of  inexperienced 
personnel  for  this  task.  The  contractor  was  thus  forced  to  rely 
on  manual  digitizing  to  complete  this  work.  The  manual  method 
involved  a "ruler  and  eyeball"  approach  to  determine  the  map 
coordinates  of  a location  or  point;  this  procedure  proved  to  be 
very  laborious  and  error  prone.  Errors  made  in  the  initial 
geographic  digitizing  effort  were  corrected  over  an  extended 
period  initially  through  contractor  quality  control  checks  and 
later  by  verification  of  system  "trouble  reports"  completed  by 
agents. 
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The  entire  experience  with  digitizing  in  CCIS  implementation 
made  the  SCRTD  and  the  contractor  recognize  that  future  efforts 
should  include  a much  greater  cost  and  time  allocation  for  com- 
pletion of  this  process. 

2.4.2  Establishment  Of  The  Transit  Data  Base 

The  data  elements  required  to  describe  the  operating 
characteristics  of  the  SCRTD  bus  service  included  the  following: 

• Bus  routes 

• Bus  stops  and  fare  zones 

• Bus  operating  schedules 

• Bus  fares 

The  level  of  difficulty  experienced  by  the  contractor  in  creating 
this  transit  data  base  was  exacerbated  by  three  technical  and 
institutional  factors.  First,  there  was  no  centralized  SCRTD 
transit  data  base;  each  of  the  above  elements  fell  under  the 
jurisdiction  of  a different  SCRTD  department.  Second,  there  was 
no  standard  format  for  the  transit  data  required  for  the  CCIS 
data  base.  SCRTD  departments  also  used  different  (IBM)  data 
processing  system  hardware,  thus  precluding  automatic  data  entry 
into  the  mainframe  computer  (Univac)  supporting  the  CCIS. 

Finally,  interdepartmental  cooperation  was  difficult  to  achieve 
since  the  potential  benefits  of  the  CCIS  data  base  were  not  fully 
understood  by  other  departments  at  this  early  stage  in  the  imple- 
mentation process. 

The  geographic  pattern  of  SCRTD  bus  service  in  the  Valley  was 
determined  by  the  32  bi-directional  bus  routes  and  the  associated 
network  of  1,800  bus  stops.  Bus  routes  were  under  the  jurisdiction 
of  the  Planning  Department  while  bus  stops  were  handled  by  the 
Stops  and  Zones  Department.  Bus  routes  were  described  in  printed 
reference  material  while  the  master  list  of  stops  and  zones  was 
contained  in  a word  processor  (VYDEC)  listing. 

The  contractor  used  these  source  data  to  integrate  the  bus 
route-bus  stop  network  into  the  CCIS  data  base.  Routes  and  stops 
were  digitized  using  the  flat  bed  system  located  at  UCLA;  this 
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effort  was  labor-intensive  and  similar  to  the  street  network 
digitizing  task.  Due  to  recurring  accuracy  problems  the  con- 
tractor actually  re-digitized  the  entire  stops  and  zones  list 
using  the  manual  method.  Bus  stop  additions  and  changes  also 
required  additional  manual  digitizing  throughout  the  implemen- 
tation period. 

Bus  operating  schedules  at  the  SCRTD  were  maintained  by  the 
Scheduling  Department.  The  schedule  data  were  stored  on  tapes 
known  as  the  B.O.S.  (Basic  Operating  Schedules)  and  processed  on 
an  IBM  1401  computer.  The  B.O.S.  system  printed  out  time  tables 
on  bus  operations  which  delineated  specific  time  points  for  each 
bus  run.  These  time  tables  were  produced  for  human  use  and  were 
not  directly  computer  usable.  Consequently,  bus  schedule  data 
were  integrated  into  the  CCIS  data  base  by  reading  the  B.O.S. 
tapes  rather  than  the  hard  copy  schedules,  directly  into  the 
mainframe  (Univac  1106)  computer.  This  integration  was  a lengthy 
task  due  to  data  and  tape  editing  requirements.  One  important 
CCIS  schedule  data  subtask  related  to  the  need  to  interpolate  bus 
run  time  points  in  those  cases  where  explicit  data  were  not 
provided.  This  effort  required  calculations  of  bus  speed  on  the 
particular  run  in  question.  Speed  calculations  varied  according 
to  the  type  and  time  of  the  day  of  the  bus  service,  e.g.,  peak 
vs.  off  peak,  day  vs.  night,  surface  vs.  freeway,  etc. 

After  entry  of  the  B.O.S.  tapes,  the  main  schedule  file  was 
stored  in  the  mainframe  computer.  Individual  schedule  changes  or 
corrections  could  then  be  made  using  a remote  terminal.  More 
extensive  bus  system  scheduling  changes  (which  occur  every  few 
months)  required  reading  an  entire  new  schedule  tape  into  the 
mainframe  computer,  superseding  the  old  tape  information.  Since 
the  pilot  CCIS  was  only  required  to  be  able  to  provide  informa- 
tion on  current  bus  service  (and  not  planned  service)  it  was 
important  for  the  implementation  team  to  coordinate  closely  with 
the  Scheduling  Department  on  the  use  of  the  correct  B.O.S.  The 
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final  element  in  the  CCIS  transit  data  base  was  the  fare 
structure.  This  fare  information  was  obtained  from  printed  SCRTD 
reference  material  and  entered  directly  into  the  CCIS  data  base 
using  a terminal. 

It  is  estimated  that  development  of  all  data  inputs  for  the 
CCIS,  including  the  geographic  and  transit  data  bases,  cost  the 
SCRTD  approximately  $50,000. 


2.5  USER  PARTICIPATION  IN  SYSTEM  DESIGN  AND  FINE  TUNING 
The  technical  foundation  of  the  CCIS  was  based  on  the  conversion 
of  the  PARIS  programs  for  application  to  the  SCRTD ' s mainframe 
computer,  and  the  creation  of  the  geographic  and  transit  data 
bases  for  the  San  Fernando  Valley.  By  November  1978  substantial 
progress  had  been  made  in  both  these  areas.  The  SCRTD  had  also 
received  delivery  of  the  computer  terminals  for  the  CCIS;  there 
were  eight  cathode  ray  tubes  supplied  for  the  pilot  system,  and 
the  contractor  was  able  to  develop  sample  format  and  response 
screens  for  system  operation.  Hardware  and  other  system  improve- 
ments cost  the  SCRTD  an  additional  $50,000. 

The  implementation  effort  now  moved  into  the  phase  of  designing 
system  operating  characteristics  which  would  satisfy  user  opera- 
tional requirements.  This  design  was  a critical  phase  in  project 
implementation  since  systems  personnel  faced  a significant  tech- 
nical challenge  while  user  personnel  were  still  quite  skeptical 
over  the  ability  of  the  system  to  actually  work.  In  fact,  several 
of  the  senior  user  staff  were  convinced  at  this  point  that  the 
system  could  never  be  developed  to  fully  satisfy  user  requirements. 
To  achieve  successful  system  design,  it  was  essential  for  the 
computer  team  to  get  users  to  specify  in  detail  their  operational 
requirements.  Given  this  skepticism,  the  standard  design 
procedure  over  the  next  few  months  was  as  follows:  the  imple- 
mentation team  would  propose  a system  feature  (e.g.,  a certain 
format  or  response  screen)  to  the  user;  typical  user  reaction  was 


-23- 


"that  doesn't  work"  or  "that's  unsatisfactory"  followed  by  "but 
this  is  what  you  really  need  to  do  to  meet  our  needs."  The  nature 
of  this  user  response  was  significant  in  that  despite  the  skepti- 
cism, it  did  specify  a given  solution.  In  this  manner  the  users 
became  a very  effective,  although  somewhat  reluctant  partner  in 
an  incremental  system  design  process.  In  the  Telephone  Infor- 
mation Section,  the  Chief  Instructor  and  her  supervisors  were 
particularly  helpful  in  making  this  operational  dialogue  worth- 
while. 

The  CCIS  route-finding  algorithms  allow  an  agent  to  specify 
either  a minimum  travel  time  route  or  a minimum  bus  transfer 
route  for  the  purpose  of  ranking  the  bus  itinerary  options  which 
would  satisfy  a given  trip  request.  The  system  technicians  and 
users  became  involved  in  a process  of  determining  the  relative 
customer  demand  for  these  two  alternative  outputs.  System 
itinerary  rankings  were  initially  based  on  minimum  travel  time 
and  weighed  all  travel  time  options  equally  with  the  exception  of 
a five-minute  penalty  for  a transfer.  Subsequent  weighing  schemes 
worked  out  with  agents  and  supervisors  increased  the  penalty  for 
a transfer  from  a minimum  of  30  to  a maximum  of  45  minutes.  This 
penalty  reflected  a long-standing  Information  Section  policy  of 
attempting  to  serve  patrons  with  only  one  bus  whenever  possible. 

It  also  reflected  operational  considerations  regarding  the  routes 
of  two  heavily  used  "horseshoe"  shaped  bus  lines  serving  the 
Valley  area. 

Users  and  system  technicians  also  became  involved  in  a process 
of  refining  machine  selection  of  appropriate  bus  stops.  Given 
the  caller's  desired  origin  and  destination  points,  the  machine 
was  programmed  to  "scan"  an  area  within  a specified  radius  for 
any  stops  on  bus  routes  appropriate  to  the  caller's  itinerary. 
Initially,  this  radius  was  set  at  one-half  mile  from  these 
points,  under  the  assumption  that  most  riders  would  be  willing  to 
walk  this  distance.  This  radius  was  decreased  to  one-quarter 
mile  at  the  request  of  the  supervisors. 
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System  designers  also  modified  the  CCIS  stop- select ion  capa- 
bilities, at  the  request  of  the  users,  to  include  a routine  for 
identifying  the  nearest  bus  stop  to  an  origin  or  destination 
point  on  a user- specif ied  bus  line.  Most  agents  could  select 
appropriate  bus  lines  for  a given  itinerary  from  memory,  but  were 
forced  to  consult  system  maps  (Thomas  Guides)  for  stop  location 
information.  With  this  computer  enhancement,  this  added  manual 
consultation  was  no  longer  necessary. 

Supervisory  personnel  became  involved  in  this  system  refinement 
process  in  January  1979.  A small  group  of  volunteers  was 
recruited  to  operate  the  evolving  system  and  provide  additional 
feedback  on  user  requirements.  Weekly  meetings  were  held  between 
agents  and  designers  to  demonstrate  system  status  and  solicit 
recommendations  for  future  improvements.  Agent  ability  to 
specify  very  practical  day-to-day  requirements  resulted  in  the 
development  of  several  valuable  system  features,  such  as  system 
specification  of  the  direction  of  bus  travel  along  the  street 
route,  an  especially  useful  feature  with  the  "horseshoe"  shaped 
bus  routes. 

Interaction  between  the  user  and  the  system  personnel 
identified  many  additional  technical  problems  to  be  resolved. 

User  abilities  to  identify  problems  and  contractor's  ability  to 
formulate  feasible  solutions  to  them  proved  to  be  the  key  to 
continuing  progress  in  system  design.  In  this  process,  system 
operating  characteristics  were  increasingly  correlated  with  user 
specifications,  and  the  CCIS  gradually  gained  more  credibility 
with  the  agents  as  an  information  retrieval  tool. 

2.6  DEVELOPMENT  OF  THE  AGENT  TRAINING  PROGRAM 

The  development  of  the  CCIS  agent  training  program  necessarily 
followed  the  system  design  process  described  above,  which  gave 
the  CCIS  its  technical  and  operational  definition.  By  March 
1979,  the  system  was  considered  sufficiently  developed  to  begin 


-25- 


training  some  regular  agents  in  the  work  force.  This  training 
effort  faced  several  difficulties.  First,  since  the  technology 
was  the  first  of  its  type  to  be  implemented  at  any  transit 
authority,  an  agent  training  program  had  not  been  attempted 
previously.  The  absence  of  an  industry  track  record  in  this  area 
left  the  implementation  team  without  any  guidelines  to  follow. 
Second,  the  major  instructional  expertise  at  this  point  resided 
with  the  contractor  team  since  they  had  developed  the  system;  an 
official  training  course  could  not  be  developed  by  the  SCRTD  this 
early  in  the  project.  Third,  operation  of  the  system  involved  a 
major  change  in  agent  work  habits,  involving  typing,  keyboard 
manipulation  and  reading  of  response  screens  from  the  CRT.  Fourth, 
the  system  data  bases  were  not  fully  refined  at  this  date.  Fifth, 
many  of  the  agents,  whose  average  education  level  was  that  of 
high  school  graduate,  were  understandably  apprehensive  about 
learning  how  to  operate  this  new  technology.  As  a result,  only  a 
few  in  the  user  group,  including  agents  and  senior  staff,  believed 
that  the  CCIS  could  really  perform  up  to  the  required  standards 
for  day-to-day  usage.  Given  this  set  of  circumstances,  the  agent 
training  program  was  initiated  by  requesting  volunteers.  These 
volunteers  were  organized  into  small  groups  for  instructional 
purposes.  Contractor  staff  conducted  the  initial  training  effort 
which  consisted  of  8 hours  of  classroom  instruction  and  8 hours 
of  terminal  practice.  The  contractor  prepared  a "CCIS  Users 
Guide  and  Training  Manual"  for  the  use  of  the  agents.  The 
training  course  was  soon  reduced  to  4 hours  of  instruction  and  4 
hours  of  terminal  operation;  this  change  was  made  based  on  the 
recommendation  of  some  senior  agents  who  considered  the  16-hour 
course  unnecessarily  lengthy;  all  the  agents  trained  in  CCIS  use 
at  this  point  were  staff  who  had  successfully  completed  the 

SCRTD' s eight-week  agent  orientation  program.  A total  of  32 
agents  initially  received  training  from  the  contractor. 


-26- 


SCRTD  training  staff,  which  until  this  point  had  taken 
responsibility  for  all  training  functions,  became  involved  with 
the  CCIS  through  the  offering  of  a short  "refresher"  course  for 
the  initial  trainees.  By  June  1980,  the  balance  of  the  agent 
work  force  (approximately  45  more  personnel)  had  been  trained  by 
SCRTD  staff  using  the  8-hour  program  developed  by  the  contractor. 

The  effectiveness  of  this  initial  training  effort  was  diffi- 
cult to  assess  at  the  time.  More  light  was  shed  on  this  area  as 
a result  of  agent  interviews  and  opinion  surveys,  discussed  in 
Section  3.5  below,  conducted  in  1980  and  1981.  There  were  clearly 
a few  agents  who  benefited  from  the  initial  training  and  who  were 
highly  motivated  to  use  and  master  this  technology;  conversely, 
there  were  some  who  reacted  quite  negatively  and  wanted  no  part 
of  the  CCIS.  Most  agents,  however,  fell  between  these  two 
extremes.  They  felt  that  they  had  been  exposed  to  a promising 
technology  but  not  subjected  to  an  intensive  enough  operational 
instruction  course  to  make  effective  use  of  it. 


2.7  SYSTEM  ACCEPTANCE 

Approximately  one  year  after  the  contract  award,  the  CCIS  was 
operational  and  could  perform  according  to  the  minimum  require- 
ments specified  in  the  RFP.  The  system  data  base  had  been 
created  (although  it  remained  error-prone  in  some  areas),  the 
information  retrieval  programs  were  functioning,  and  transaction 
response  time  was  within  the  standard  specified.  It  was  evident 
at  this  point,  however,  that  a longer  "debugging"  and  refinement 
period  would  be  required  to  bring  the  system  to  the  point  where 
users  had  confidence  in  its  operation. 

The  SCRTD  officially  accepted  the  CCIS  on  June  4,  1979  in 
accordance  with  the  terms  of  the  contract. 


2.8  SYSTEM  IMPROVEMENTS  AND  USER  ACCEPTANCE  (1979-1980) 

After  system  acceptance,  operational  deployment  of  the  CCIS  in 
the  SCRTD  Telephone  Information  Center  consisted  of  seven  regular 
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agent  work  positions  equipped  with  CRT  terminal  units,  with  an 
additional  unit  used  for  system  updating.  Each  position  was 
provided  with  telephone  equipment  capable  of  answering  calls  on 
both  the  San  Fernando  Valley  area  (Van  Nuys)  and  the  Downtown  Los 
Angeles  Area  trunk  lines.  Because  CCIS  capabilities  were  limited 
to  the  Valley  area,  agents  were  instructed  to  place  high  priority 
on  Van  Nuys  trunk  calls,  and  handle  calls  on  the  downtown  trunk 
only  if  there  were  no  Van  Nuys  calls,  a circumstance  which 
occurred  only  rarely.  In  order  to  answer  downtown  area  trunk 
calls,  as  well  as  any  calls  for  itineraries  originating  in  the 
Valley,  but  traveling  to  destinations  outside  the  CCIS's  geo- 
graphic scope,  agent  work  positions  were  also  equipped  with  all 
the  indexed  schedule  materials  available  for  manual  data 
retrieval.  Agents  were  encouraged  to  use  the  CCIS  as  much  as 
possible,  but  to  double  check  any  suspicious  response  screens 
against  the  manual  hard  copy  data. 

The  first  year  of  CCIS  operation  was  actually  a continuation 
of  the  system  implementation  "shakedown"  period  in  which  agent 
experiences  with  CCIS  call  processing  resulted  in  identification 
of  numerous  areas  where  further  improvements  were  required. 
Technical  improvement  efforts  by  the  system  contractor  included 
the  following: 

• Corrective  actions  - where  errors  in  system  data  base  or 
operations  were  detected  and  corrected. 

• System  refinements  - where  improvements  were  made  to 
existing  system  operating  features. 

• System  enhancements  - where  new  technical  features  were 
added  to  improve  overall  system  performance. 

Corrective  actions  were  greatly  facilitated  through  establishment 
of  a trouble  reporting  system.  When  there  was  a question  con- 
cerning the  validity  of  any  CCIS  response  screen,  it  was  the 
agent's  responsibility  to  verify  manually  the  suspected  error  and 
to  document  this  discrepancy  on  a trouble  report.  These  reports 
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were  submitted  to  shift  supervisors,  who  subsequently  turned  them 
over  to  the  system  contractor  for  verification  and/or  correction. 
Examples  of  errors  detected  and  reported  included  schedule  data 
inconsistencies,  wrong  bus  stop  locations,  improperly  digitized 
street  addresses  or  landmarks,  and  "bizarre"  bus  itinerary 
responses. 

As  might  be  expected  the  initial  number  of  trouble  reports  was 
quite  high;  this  situation  was  due  to  the  newness  of  the  system 
and  the  fact  that  multiple  reports  of  the  same  erroneous  data 
were  made.  A trouble  report  response  procedure  was  established 
whereby  each  agent  submitting  a report  was  provided  with  an 
answer  explaining  the  discrepancy  observed  and  the  corrective 
action  taken,  if  any.  Interestingly,  there  were  several  cases  in 
which  the  perceived  "bizarre"  bus  routing  reported  was  actually 
deemed  the  "best"  routing  in  light  of  the  query  input.  Such 
cases  were  valuable  in  that  they  demonstrated  the  educational 
potential  of  the  system,  even  for  experienced  agents.  Review  of 
these  cases  resulted  either  in  a change  of  the  agent's  compre- 
hension of  the  system,  or  in  a modification  by  the  contractor  of 
the  various  weighting  factors  in  the  system's  route- f ind ing 
algorithm.  By  mid-1980,  there  were  very  few  trouble  reports 
being  submitted  by  the  agents,  indicating  that  the  system  was 
approaching  a relatively  high  degree  of  accuracy. 

Refinements  to  the  system  focused  on  improving  system  operating 
features  with  the  objective  of  promoting  more  effective  system 
usage  by  the  agents.  Areas  that  received  improvements  included 
the  following: 

• Estimation  through  interpolation  of  schedule  time  points, 
using  realistic  bus  speeds. 

• Better  identification  of  streets  traveled  along  a given 
bus  route,  and  direction  of  travel  along  those  streets  at 
given  times. 
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• Integration  into  the  system  of  school  bus  trips,  which 
use  the  same  route  and  line  numbers  as  regular  buses. 

• Continued  intensive  study  of  weighting  factors  to 
determine  rankings  for  bus  itinerary  options.  This  was 
the  most  extensive  effort  of  the  system  refinement  pro- 
cess, since  its  major  objective  was  to  enable  supervisors 
and  agents  to  better  understand  the  designer's  rationale 
behind  responses  provided. 

Enhancements  to  the  CCIS  made  during  this  improvement  period 
included : 

• The  addition  of  a "forced"  routing  transaction.  This 
transaction  enabled  agents  to  specify  a "preferred"  bus 
line  prior  to  requesting  a system  itinerary  response. 

The  system  was  then  "forced"  to  respond  with  an  itinerary 
which  included  the  particular  bus  line  requested . This 
transaction  proved  to  be  a valuable  addition  to  the  CCIS 
from  a marketing  standpoint,  as  in  many  cases,  callers 
familiar  with  a certain  route  persist  in  requesting 
information  regarding  that  route  even  when  informed  that 
it  is  not  the  "best"  available.  Forcing  the  system  to 
use  a certain  route  saved  the  agent  from  time-consuming 
back-checking  of  manual  data  and  at  the  same  time  met  the 
demands  of  the  caller. 

• Changes  to  the  schedule  request  transaction  to  allow  for 
" shor tl in ing " in  which  a certain  bus  reverses  direction 
at  the  request  of  the  dispatcher  without  completing  its 
full  route. 

• Correlation  of  geographic  locations  listed  on  response 
screens  with  the  page  number  and  map  coordinates  of  those 
locations  in  the  agent's  system  map  book  (Thomas  Guide). 
This  feature  enabled  agents  to  locate  itinerary  origin, 
transfer,  and  destination  points  much  faster  when 
necessary  and  facilitated  the  provision  of  walking 
instructions. 
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• Reduction  of  average  system  response  time  by  eliminating 
the  need  for  the  system  to  consider  certain  routing 
solutions. 

• Investigation  of  alternatives  for  providing  walking 
instructions  using  video  display  systems  with  slide 
projections  of  Thomas  Guide  maps.  Since  the  CGIS  expli- 
citly lacks  the  capacity  to  provide  walking  instructions, 
the  development  of  alternatives  to  improve  agent  pro- 
ductivity in  this  area  was  given  considerable  attention. 

The  technical  improvement  period  resulted  in  a more  accurate 
data  base,  more  user  features,  and  more  responsive  system 
operation.  It  raised  the  CGIS  to  a level  of  technical  maturity 
where  the  user  could  comfortably  and  effectively  operate  the 
system.  At  this  point,  further  system  progress  depended  more 
upon  increased  and  managerial  support  within  the  SGRTD. 

2.9  EVALUATION  GONTRAGTOR  REGOMMENDATIONS  IN 
SUPPORT  OF  SYSTEM  IMPLEMENTATION 

During  the  summer  of  1980,  a contractor,  Wilson  Hill  Associates, 
Inc. , was  named  by  UMTA  through  its  Transportation  Systems  Genter 
to  conduct  an  independent  socio-economic  and  performance  evalua- 
tion of  the  GGIS.  This  evaluation  was  intended  to  measure  specific 
impacts  on  user  performance  and  productivity  as  well  as  more 
general  impacts  on  the  SGRTD ' s marketing  program.  Early  assess- 
ments of  the  system  by  this  contractor  indicated  a number  of 
institutional  problems  that  were  hindering  its  use;  in  mid-1980 
agents  were  at  best  consulting  GGIS  only  intermittently.  They 
often  double-checked  response  screens  against  manual  references, 
thereby  greatly  slowing  down  their  response  times  and  decreasing 
their  productivity.  It  was  obvious  that  the  technical  improvements 
to  hardware  and  software  described  above  were  only  part  of  the 
effort  needed  to  enable  the  pilot  system  to  achieve  its  full 
potential.  User  attitudes  showed  that  the  SGRTD  also  needed  to 
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make  a stronger  commitment  to  the  human  component  of  the  system: 
the  users.  This  commitment  was  required  from  the  Telephone 
Information  Section  ^and  several  other  SCRTD  departments. 

The  SCRTD  Director  of  Consumer  Relations  worked  with  the 
Information  Section  to  implement  a set  of  six  evaluation  con- 
tractor recommendations  designed  to  promote  CCIS  usage  and 
effectiveness.  These  recommendations  included: 

• Establishment  of  a CCIS  Working  Committee  - this 
committee  established  objectives  for  system  usage, 
identified  CCIS  related  problems,  recommended  solutions 
to  identified  problems,  and  monitored  system  data  to 
ascertain  the  achievement  of  objectives.  Committee 
membership  included  senior  personnel  from  the  Telephone 
Information,  Computer  Operations,  Scheduling,  Planning, 
and  Stops  and  Zones  departments. 

• Appointment  of  a supervisor  as  full  time  CCIS  Program 
Manager  - this  program  manager  was  responsible  for 
effective  CCIS  operation  and  served  as  the  intermediary, 
between  the  CCIS  agents  and  the  system  technicians, 
effectively  "speaking  the  language"  and  satisfying  infor- 
mation needs  of  both. 

• Development  of  strategies  with  Computer  Operations  to 
improve  CCIS  response  times  - a strategy  was  designed  to 
support  responsive  CCIS  transactions  on  the  mainframe 
computer  by  scheduling  other  major  computing  jobs  (e.g., 
SCRTD  payroll  computations)  during  late-night  periods  of 
low  call  demand. 

• Creation  of  a new  program  for  CCIS  agents  involving 
re-training  and  improved  status  and  support  - all  agents 
desiring  to  work  the  CCIS  were  asked  to  volunteer  for  new 
training  which  was  followed  by  a testing  and  certifica- 
tion process;  agents  were  assigned  to  more  desirable 
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daytime  work  shifts  and  provided  with  a steady  support 
program  for  resolving  CCIS-related  problems. 

• Development  of  a program  for  greater  interdepartmental 
cooperation  in  supporting  the  CCIS  transit  data  base  - 
this  program  was  aimed  at  developing  more  effective 
procedures  for  receiving  and  updating  data  from  the 
Scheduling,  Planning,  and  Stops  and  Zones  Departments 
within  the  SCRTD. 

• Use  of  the  CCIS  log  file  system  data  collection  program 
for  analysis  of  system  and  agent  performance  - the 
contractor  expanded  the  CCIS  log  file  program  to  permit 
the  calculation  of  performance  measures  for  the  whole 
system  and  for  individual  agents;  these  data  were  used  by 
the  CCIS  Program  Manager  and  the  Working  Committee  to 
analyze  system  operations  and  evaluate  agent  progress  and 
skill  development. 

These  recommendations  were  substantially  implemented  between 
November,  1980  and  February  1981.  Their  implementation  repre- 
sented a breakthrough  in  enlisting  user  support  and  enthusiasm 
for  the  system.  In  addition,  establishment  of  the  Working  Com- 
mittee caused  other  SCRTD  departments  to  recognize  the  potential 
of  the  CCIS  data  base  as  a valuable  resource.  The  agents 
responded  positively  to  these  recommendations,  and  42  of  the  85 
agents  in  the  Telephone  Information  Department  volunteered  for 
CCIS  retraining.  By  February  22,  1981,  the  CCIS  was  considered 
ready  for  an  official  90-day  evaluation  of  system  operation, 
performance  and  impacts. 

2.10  CONCLUSIONS  REGARDING  SYSTEM  DEVELOPMENT 

In  retrospect,  it  can  be  stated  that  the  early  criterion 
established  by  the  Contractor  Selection  Committee  of  accepting 
only  "workable"  systems  or  "existing"  software  was  somewhat  of  a 
myth.  The  sheer  magnitude  of  the  effort  required  to  implement 
the  CCIS  so  totally  outclassed  SDC ' s previous  efforts  with  the 
PARIS  system  in  Santa  Monica  that  they  were  required  to  start 
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almost  from  scratch. 

It  appears  unlikely,  given  the  unique  service  networks,  as 
well  as  departmental  and  institutional  frameworks  of  most  large 
transit  authorities,  that  the  CCIS  would  be  directly  transferable 
to  other  transit  properties  without  extensive  modifications  to 
system  software.  Other  properties  can,  however,  learn  from  major 
implementation  problems  at  SCRTD  and  plan  accordingly.  For 
example : 


• Creation  of  the  geographic  data  base  requires  a very 
labor-intensive  effort  which  is  greatly  facilitated  by 
use  of  state-of-the-art  digitizing  equipment; 

• It  is  extremely  important  to  secure  and  hold  confidence 
of  system  users  from  the  outset  by  seeking  their  ideas 
and  opinions; 

• Cooperation  from  other  authority  departments  is  essential 
to  construction  of  the  transit  data  base;  and 

• A comprehensive  management  system  made  up  of  represen- 
tatives of  all  parties  (users,  contractor.  Marketing 
Department  management,  and  other  affected  departments) 
should  be  implemented  as  an  integral  part  of  the  process. 
This  system  should  have  as  its  chief  function  the  facili- 
tation of  responses  to  perceived  implementation  problems 
as  they  arise. 
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3.0  SYSTEM  EVALUATION 


With  the  full  cooperation  of  the  system  design  contractor 
and  the  management  and  staff  of  the  SCRTD  Telephone  Information 
Section,  Wilson  Hill  Associates  in  the  summer  of  1980  designed  a 
comprehensive  program  for  collection  and  evaluation  of  data 
concerning  the  CCIS.  These  data  were  collected  between  October 
1980  and  June  1981.  The  major  goals  of  this  program  were  to 
determine,  as  accurately  as  possible,  answers  to  the  following 
questions; 

• In  what  ways  does  the  CCIS  affect  the  Telephone  Infor- 
mation Section's  productivity,  i.e.,  the  overall  quantity 
of  agent  responses  provided? 

• In  what  ways  does  the  CCIS  affect  the  quality,  i.e.,  the 
accuracy  and  consistency  of  agent  responses  provided? 

• How  does  the  CCIS  affect  an  average  agent's  performance 
of  normal  job  functions?  How  does  it  change  the  agent's 
perception  of  his/her  job? 

• What  impacts  does  the  CCIS  have  on  the  training  process 
for  new  agents? 

• What  other  impacts,  positive  or  negative,  does  the  system 
exert  on  other  SCRTD  departments?  On  the  general  public? 

A number  of  different  data  collection  instruments  were 
implemented  as  elements  of  this  evaluation  program,  taking 
advantage  of  a variety  of  different  information  sources.  In  this 
section,  efforts  at  quantitative  impact  measurement  are  presented 
first;  qualitative  evaluation  efforts  are  then  described. 

3.1  QUANTITATIVE  DATA  COLLECTION 

In  evaluating  the  quantitative  impacts  of  the  CCIS,  Wilson 
Hill  had  three  primary  objectives; 

• To  determine  through  measurement  of  call  volumes  the 
characteristics  of  consumer  demands  placed  on  the  Infor- 
mation Section,  both  from  the  San  Fernando  Valley  (the 
CCIS  test  area)  and  from  the  entire  SCRTD  service  area. 
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• To  ascertain  as  accurately  as  possible  the  performance  of 
the  CCIS  agents,  as  compared  with  their  manual  counter- 
parts, in  responding  to  these  demands. 

• To  gain  a detailed  understanding  of  the  characteristics 
of  CCIS  calls. 

Four  .separate  types  of  data  were  used  in  the  achievement  of 
these  objectives.  these  were: 

• Automatic  Call  Distribution  system  counts; 

• Van  Nuys  trunk  daily  call  counts; 

• CCIS  internal  log  file  statistics;  and 

• Agent  performance  data  recorded  under  controlled  experi- 
mental conditions  set  up  by  the  evaluation  contractor. 

3.1.1  Automatic  Call  Distribution  System  Data 

The  SCRTD  uses  a S tromberg-Car 1 son  Automatic  Call  Distribu- 
tion (ACD)  system  to  route  incoming  calls  to  the  information 
agents.  This  equipment  receives  calls  on  83  separate  trunk  lines. 
When  a call  is  received,  it  is  transferred  directly  to  an  availa- 
ble agent.  If  all  agents  are  busy,  the  call  is  routed  to  a holding 
queue  from  which  calls  are  routed  in  a first-in,  first-out  sequence. 
This  ensures  that  the  caller  who  has  waited  the  longest  is  handled 
by  the  next  available  agent.  While  holding,  a caller  receives  a 
recorded  message  stating  that  all  operators  are  currently  busy 
and  requesting  that  the  caller  organize  his/her  question  before 
the  operator  comes  on  the  line. 

The  ACD  system  is  a valuable  source  of  data  concerning  both 
caller  demand  and  agent  performance.  Using  a device  called  a 
"peg  counter,"  it  mechanically  records  the  number  of  calls 
entering  the  system  and  the  number  of  calls  which  are  "lost," 
i.e.,  those  which  are  ended  by  the  caller  before  reaching  an 
agent.  On  a form  labeled  "Trunk  Group  Readings,"  incoming  calls 
and  lost  calls  are  recorded  from  the  counter  each  hour  by  SCRTD 
supervisory  personnel.  Once  these  data  are  recorded,  the  ACD 
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counter  is  reset  to  zero  for  the  next  hour's  count.  All  calls 
from  the  San  Fernando  Valley,  the  CCIS  test  area,  are  recorded  by 
this  system  in  a single  trunk  group  (the  Van  Nuys  Trunk).  This 
circumstance  facilitated  comparisons  between  caller  demands  and 
agent  performance  for  both  CCIS  and  manual  agents. 

It  might  be  noted  that  several  of  the  management  personnel 
in  the  Telephone  Information  Section  expressed  reservations 
concerning  the  accuracy  of  the  Trunk  Group  Readings.  The  ACD 
system  operating  during  the  period  of  data  collection  was 
relatively  old  and  was  experiencing  numerous  component  mal- 
functions. The  entire  system  was  replaced  with  new  equipment 
in  July  1981.  Despite  these  reservations  ACD  data  were  still 
used  quite  extensively  by  SCRTD  throughout  the  data  collection 
period  as  indicators  of  agent  performance  and  caller  demand. 


3 . ]. . 2 CCIS  Log  File  Data 

In  the  early  stages  of  CCIS  implementation,  the  SCRTD 
found  it  useful  to  monitor  CCIS  agent  progress  by  taking 
detailed  ACD  counts  of  the  number  of  calls  answered  by  each 
CCIS  agent  each  hour.  These  data  were  compiled  on  forms 
labeled  "Van  Nuys  Daily  Call  Count." 

The  CCIS  log  file,  which  superseded  the  Van  Nuys  Daily 
Call  Count  as  a data  collection  device  for  the  Van  Nuys  trunk, 
is  an  extremely  valuable  data  collection  component  which  takes 
advantage  of  the  system's  inherent  data  storage  capabilities. 

Its  development  was  an  outgrowth  of  the  SCRTD  specification 
which  required  that  the  system  be  capable  of  producing  hard- 
copy records  of  all  transactions.  Such  transaction  records 
were  potentially  useful  for  analysis  but  rather  unwieldy  in 
terms  of  length  and  detail.  In  consultations  with  SDC  and  the 
data  processing  staff  at  SCRTD,  a computer  program  was  developed 
which  automatically  produced  a daily  compilation  of  certain 
system  statistics.  This  program  was  a modification  of  a data 
reduction  and  report  production  program  developed  earlier  by 
SDC  for  its  Wn  use. 
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The  CCIS  log  file  includes  information  on  all  aspects  of 
system  performance.  In  most  instances  this  information  is 
presented  both  for  each  individual  agent  and  for  the  entire 
system  (usually  seven  agents).  Log  file  results  are  compiled 
during  CCIS  operational  hours  (6  am  to  6 pm)  and  a record  is 
kept  of  each  day's  transactions. 

The  accuracy  of  the  CCIS  log  file  was  independently 
established  by  random  spot  samples  of  calls  conducted  by  the 
evaluation  contractor,  using  monitoring  equipment  available  at 
the  Telephone  Information  Section.  By  eliminating  human  error 
in  the  data  collection  process,  the  CCIS  log  file  provided  a 
complete  and  detailed  history  of  CCIS  operations.  Data  items 
recorded  on  the  CCIS  log  file  are  listed  in  Table  3-1. 

One  drawback  of  the  CCIS  log  file  is  that  data  are  compiled 
only  for  those  calls  which  are  answered  using  CCIS  equipment. 

No  records  are  kept  of  total  calls  or  lost  calls  on  the  Van 
Nuys  trunk.  For  this  information,  the  evaluation  contractor 
was  forced  to  rely  on  the  less  accurate  Trunk  Group  Readings 
from  the  ACD  system. 


3.1.3  CCIS  Controlled  Experiment  Data 

While  the  above-mentioned  sources  offered  a wealth  of 
information  concerning  various  aspects  of  Telephone  Information 
Section  operations,  they  still  did  not  provide  a completely 
unbiased  comparison  of  performance  between  the  CCIS  and  manual 
modes  of  information  retrieval  for  impact  analysis.  Several 
factors  inherent  to  the  system  created  comparison  problems: 

• Those  agents  trained  for  CCIS  are  generally  more 

experienced  than  the  average  agent.  This  situation 
is  due  to  the  fact  that  the  CCIS  was  operational 
during  the  day  shift  (6  am  - 6 pm) , the  shift  most 
preferred  by  Section  employees.  Since  shift  assign- 
ments are  made  on  the  basis  of  union  seniority. 
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TABLE  3-1.  CCIS  LOG  FILE  DATA 
COMPILED  DAILY 


OPERATOR  DATA: 
CALL  DATA: 


CALL  TIME  DATA 


Agent  ID  numbers  and  logon/logoff  times 

Call  count  by  terminal,  total  calls 

Manual  call  count  by  terminal,  total 
manual  calls 

CCIS  call  count  by  terminal,  total  CCIS 
calls 

Number  of  calls  by  hours  of  day,  6 am-5  pm 

Average  call  time  by  starting  hour 

Average  call  time  by  terminal,  CCIS  and 
manual 

Average  CCIS  call  time  by  terminal 

Average  manual  call  time  by  terminal 

Call  count  by  classes  of  call  duration 

Percent  of  calls  in  each  duration  class 

Overall  average  call  time 

Overall  standard  deviation  of  call  times 

Projected  calls  per  hour  by  terminal, 
manual  and  CCIS 

Projected  calls  per  hour  by  terminal, 
manual  calls  only 

Projected  calls  per  hour  by  terminal, 

CCIS  calls  only 


TRANSACTION  DATA 


Total  transactions 
Processing  transactions 


Transaction 

count 

by 

type 

Transaction 

count 

by 

termi 

Transaction 

count 

by 

hour 

Transaction 

count 

by 

type 

Computed  it 

iner ar i 

es 

Return  itineraries  computed 
Transactions  per  call 
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TABLE  3-1.  Continued... 


RETRIEVAL 
TIME  DATA: 


ERROR  DATA 


OTHER  CCIS 


Retrieval  count  by  duration  class,  all 
transactions 

Retrieval  % by  class,  all  transactions 

Retrieval  count  by  class,  computed  itineraries 

Retrieval  % by  class,  computed  itineraries 

Retrieval  count  by  class,  all  but  computed 
itineraries 

Retrieval  % by  class,  all  but  computed 
itineraries 

Average  retrieval  time 

Standard  deviation  of  retrieval  time 

Total  errors 

User  errors  by  terminal 

System  software  errors  by  terminal 

System  hardware  errors  by  terminal 

DATA:  Average  computer  time  per  CCIS  call 

% of  CCIS  calls  used  in  retrieval 

Average  time  from  beginning  of  call  to 
first  transaction 

Average  time  from  last  transaction  to 
end  of  call 
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those  CCIS-trained  agents  having  longer  job  tenure 
generally  elect  and  receive  CCIS  duty. 

The  configuration  of  the  ACD  hardware  which  routes 
calls  to  agents  generally  ensures  a shorter  queue  on 
the  Van  Nuys  trunk  line  than  on  the  other  trunks. 

As  a result,  callers  usually  experience  a shorter 
holding  period  on  the  Van  Nuys  trunk  than  on  the 
others.  It  is  likely  that  this  shorter  wait  is 
reflected  in  fewer  lost  calls  on  the  Van  Nuys  trunk. 
CCIS  agents,  during  normal  operations,  do  not  exclu- 
sively use  the  computer  for  data  retrieval.  Because 
the  CCIS  data  base  contains  information  on  only  a 
portion  of  the  total  SCRTD  service  area,  agents 
occasionally  receive  requests  for  routings  to  des- 
tinations outside  the  system's  geographic  limits. 

To  handle  these  requests,  agents  must  resort  to 
manual  retrieval.  In  addition,  on  rare  occasions 
when  there  are  no  incoming  calls  on  the  Van  Nuys 
trunk  agents  are  instructed  to  answer  calls  origi- 
nating on  the  downtown  Los  Angeles  trunk.  Handling 
these  calls  implies  manual  retrieval,  since  the 
caller's  origin  is  also  outside  the  system's  scope. 
Even  in  instances  where  the  CCIS  can  handle  both 
origin  and  destination  data,  agents  often  double- 
check response  screens  which  they  deem  questionable 
against  the  manual  data. 

The  calls  handled  by  the  Information  Section  are 
themselves  inherently  variable.  One  caller  could 
have  a single  request  that  required  a 3-second 
response,  while  another  might  have  multiple 
itinerary  queries  that  demanded  up  to  10  or  15 
minutes'  response  time. 
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As  a result  of  these  factors,  the  evaluation  contractor 
realized  that  a direct  comparison  of  data  retrieval  modes 
based  only  on  system  operations  data  would  not  be  possible.  In 
order  to  make  such  a direct  comparison  in  which  bias  was 
eliminated,  an  artificial  environment  was  required  in  which 
all  of  the  above-mentioned  factors  were  controlled.  With  the 
assistance  of  the  SCRTD,  the  contractor  created  such  a con- 
trolled experimental  setting  in  November  1980.  In  this 
setting,  nine  SCRTD  agents  were  tested.  Each  agent  was 
queried  with  the  same  set  of  36  itinerary  questions  concerning 
transit  service  in  the  CCIS  service  area.  These  questions 
were  developed  by  the  Information  Section  with  the  assistance 
of  evaluation  contractor  personnel.  They  were  designed  to 
test  both  the  capabilities  of  the  agents  and  the  system.  The 
nine  agents  represented  three  modes  of  information  retrieval 
and  three  levels  of  agent  skill  and  experience.  The  experiment 
was  conducted  by  two  members  of  the  contractor  evaluation 
team,  who  alternated  in  performing  the  roles  of  caller  and 
observer.  The  caller  made  the  test  calls  from  a separate  room 
and  recorded  the  agent  responses  on  a log  form.  The  observer 
was  located  in  the  same  room  as  the  test  agent,  but  out  of 
direct  view;  total  call  times  were  logged  by  the  observer 
using  a stop  watch.  Agent  responses  and  response  speeds  for 
the  various  retrieval  modes  and  skill  levels  were  later  compared 
and  evaluated. 

3.2  QUANTITATIVE  EVALUATION  METHODOLOGY 

The  wide  variety  of  quantitative  data  sources  described 
above  necessitated  the  development  of  a number  of  different 
evaluation  strategies. 
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For  the  ACD,  Van  Nuys  trunk,  and  CCIS  log  file  data,  it 
was  decided  that  a 90-day  period  of  data  collection  during 
steady-state  operation  of  the  CCIS  would  provide  an  adequate 
information  base  for  impact  evaluation.  This  period  was  con- 
sidered to  be  of  sufficient  duration  to  average  out  any  observed 
extremes  in  data  variability,  and  to  ascertain  general  trends, 
if  any,  in  variables  studied.  The  mode  of  evaluation  used  for 
these  three  data  sources  might  therefore  best  be  termed  "inter- 
rupted time-series  with  a nonequivalent  control  group."  In 
essence,  both  the  experimental  group  (the  CCIS  agents)  and 
the  control  group  (agents  on  trunk  other  than  Van  Nuys)  were 
measured  repeatedly  over  the  90-day  evaluation  period. 
"Interrupted"  merely  signifies  that  data  collection  was  not 
continuous,  since  the  CCIS  log  file  was  operational  only  between 
6 am  and  5 pm.  This  experimental  design  is  considered  one  of 
the  best  available  for  eliminating  extraneous  factors  as  explana- 
tions for  observed  changes.* 

Not  all  of  the  measures  collected  in  the  90-day  evaluation 
period  fit  into  this  experimental  design.  In  fact,  most  of 
the  variables  compiled  on  the  CCIS  log  file  could  not  be 
obtained  for  the  manual  agents;  for  these  variables  the 
experimental  methodology  might  best  be  called  "interrupted 
time-series  without  control  group."** 


* Mark  S.  Sanders  and  Denis  J.  Sullivan,  An  Experimental  Design 
Plan  of  SCRTD  Computer  Customer  Information  Service,  Canyon 
Research  Corp. , Dec.  1977,  p.8. 

**  Ibid. , p.  9 . 
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In  the  case  of  the  controlled  experiment,  extended  data 
collection  was  not  possible  due  to  logistical  reasons.  During 
the  course  of  this  experiment,  a total  of  324  agent  responses 
were  recorded  and  timed.  Experimental  conditions  were  suffi- 
ciently controlled  so  that  an  analysis  of  variance  could  be 
performed  on  these  observed  response  times.  The  analysis  of 
variance  is  a statistical  method  of  dividing  the  variation 
observed  in  experimental  data  into  different  parts,  each  part 
assignable  to  a known  source  or  cause.  It  allows  the  evaluator 
to  assess  the  relative  magnitudes  of  variation  that  are  the 
result  of  each  of  the  independent  variables,  which  in  this 
case  were  agent  experience  (novice,  intermediate,  or  advanced) 
and  mode  of  data  retrieval  (CCIS  only,  manual  only,  or  a 
mixture  of  manual  and  CCIS) . 

3.3  QUANTITATIVE  EVALUATION  RESULTS 

The  analyses  presented  below  are  based  on  the  following 
data  made  available  to  the  evaluation  contractor  by  the  SCRTD: 

• Van  Nuys  ACD  daily  call  count  forms  covering  the 
period  from  November  23,  1980  to  February  22,  1981. 

• CCIS  log  file  printouts  covering  the  period  from 
February  22,  1981  to  June  4,  1981.  Although  the 
system  was  functioning  continuously  during  this 
period,  log  file  data  for  March  10,  March  31,  April 
26  through  May  2,  and  May  22  were  not  made  available 
to  the  contractor. 

• ACD  Trunk  Group  Reading  forms  covering  the  period 
from  February  21  to  May  22,  1981. 

Based  on  the  data  provided,  it  was  decided  that  the  90-day 
evaluation  period  would  run  from  February  22  to  May  22,  1981. 
Due  to  the  missing  week  of  CCIS  log  file  data  at  the  end  of 
April,  most  log  file-based  trend  analyses  were  extended  an 
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extra  week  to  May  30,  1981.  The  CCIS  Controlled  Experiment  is 
the  subject  of  a companion  report  prepared  by  Wilson  Hill.* 

Since  the  companion  report  contains  detailed  analyses  of  experi- 
mental results,  only  the  most  salient  conclusions  on  agent 
call  times  will  be  presented  in  this  report.  These  conclusions 
follow  the  descriptions  of  caller  demand,  agent  performance, 
and  call  characteristics  below. 

3.3.1  Characteristics  of  Caller  Demand 

Using  the  ACD  data,  all  calls  received  by  the  Telephone 
Information  Section  on  all  trunks  between  6 am  and  5 pm  were 
totaled  for  each  day  of  the  90-day  period.  Totals  for  this 
time  span  for  each  day  in  the  test  period  are  presented  in 
Appendix  A.  Calls  received  on  all  trunks  during  this  time 
period  ranged  from  a low  of  4336  on  Sunday,  March  1 to  a high 
of  9382  on  Monday,  April  7.  The  average  number  of  calls 
received  daily  on  all  trunks  for  the  entire  test  period 
between  6 am  and  5 pm  was  6723;  the  standard  deviation  was 
1413. 

Total  calls  received  by  the  section  were  averaged  for 
each  week  of  the  test  period;  the  results  are  shown  in  Figure 
3-1.  From  Figure  3-1,  it  can  be  seen  that  caller  demand  varies 
considerably  from  week  to  week.  The  first  6 weeks  of  the  test 
period  were  characterized  by  relatively  high  caller  demand, 
with  calls  averaging  7134  between  6 am  and  5 pm.  The  next  7 
weeks,  beginning  April  5,  show  appreciably  lower  demand,  with 
an  average  6 am  to  5 pm  count  of  6370. 

Of  particular  interest  in  the  ACD  data  were  calls  received 
by  the  Van  Nuys  Trunk  serving  the  CCIS.  These  readings  were 
totaled  for  the  hours  6 am  to  5 pm  for  each  day  of  the  test 
period;  these  total  calls  are  shown  in  Appendix  B.  Calls 


* Robert  E.  Furniss  and  Robert  0.  Phillips,  The  CCIS  Experiment: 
Comparing  Transit  Information  Retrieval  Modes  at  the  SCRTD, 
Wilson  Hill  'Associates 
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TOTAL  CALLS  RECEIVED,  ALL  TRUNKS 
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FIGURE  3-1.  TOTAL  DAILY  CALLS  RECEIVED 
BY  TELEPHONE  INFORMATION  SECTION: 
AVERAGE  FOR  EACH  WEEK  OF  THE  TEST  PERIOD 

(6 am  — 5 pm) 


Source:  ACD  Trunk  Group  Readings 
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received  on  the  Van  Nuys  Trunk  during  the  6 am  to  5 pm  time 
period  ranged  from  a low  of  560  on  March  22  to  a high  of  1549 
on  April  14.  The  average  number  of  calls  received  on  this 
trunk  between  6 am  and  5 pm  was  972;  the  standard  deviation 
was  200. 

As  in  Figure  3-1,  total  calls  received  on  the  Van  Nuys 
trunk  were  averaged  for  each  week  of  the  test  period;  the 
results  are  shown  in  Figure  3-2.  From  Figure  3-2,  it  can  be 
seen  that  the  weekly  patterns  of  Van  Nuys  caller  demand  are 
quite  independent  of  those  for  the  entire  facility.  The 
correlation  coefficient  r^  between  the  two  demand  distribu- 
tions is  only  .31,  indicating  very  low  correlation  between 
them.  Unlike  the  pattern  shown  in  Figure  3-1,  Van  Nuys 
demand  was  relatively  low  for  the  first  6 weeks  of  the  test 
period,  averaging  approximately  903  calls  per  day.  For  the 
last  7 weeks  of  the  test  period,  however.  Van  Nuys  demand 
increased  considerably,  especially  during  the  weeks  of  April 
12-18  and  May  10-16.  The  average  daily  demand  on  the  Van  Nuys 
trunk  for  these  last  7 weeks  was  1030  calls. 

Figure  3-3  presents  the  result  obtained  when  the  average 
daily  van  Nuys  demand  is  divided  into  average  daily  total  demand 
for  each  week  of  the  test  period.  Figure  3-3  thus  represents 
the  average  daily  market  share  of  the  Van  Nuys  trunk  for  each 
week  of  the  test  period.  As  can  be  seen  from  Figure,  3-3  the 
Van  Nuys  trunk  accounted  for  slightly  under  15%  of  all  calls  on 
average  during  the  test  period.  The  average  Van  Nuys  trunk 
market  share  fluctuated  from  a low  of  11.3%  of  all  calls  during 
the  week  of  March  29  - April  4 to  a high  of  19.7%  during  the 
week  of  April  12  - 18.  The  first  6 weeks  of  the  test  period 
were  characterized  by  a relatively  low  (12.8%)  Van  Nuys  market 
share.  During  the  final  7 weeks  of  the  test  period,  the  Van 
Nuys  share  rose  appreciably  to  16.3%  of  all  calls. 
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FIGURE  3-2.  TOTAL  DAILY  CALLS  RECEIVED, 
VAN  NUYS  (CCIS)  TRUNK  AVERAGE  FOR  EACH  WEEK 
OF  THE  TEST  PERIOD 
(5  am  — 5 pm) 


Source:  ACD  Trunk  Group  Readings 
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PERCENT  OF  ALL  INCOMING  CALLS 


WEEK 


FIGURE  3-3.  DAILY  MARKET  SHARE  OF  VAN  NUYS  (CCIS)  TRUNK: 
AVERAGE  FOR  EACH  WEEK  OF  THE  TEST  PERIOD 

(6  am  — 5 pm) 


Source:  ACI^  Trunk  Group  Readings 
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Figures  3-4  and  3-5  show  results  obtained  by  averaging 
total  calls  received  and  Van  Nuys  trunk  calls  received  between 
6 am  and  5 pm  by  day  of  the  week  over  the  entire  period.  The 
patterns  shown  on  Figures  3-4  and  3-5  are  similar;  they  show 
that  caller  demands  exhibit  a high  degree  of  fluctuation  during 
the  week,  a fact  that  has  strong  implications  on  scheduling  of 
Information  Section  staffing.  In  both  figures,  Sunday  demand 
is  the  lowest  for  any  day  of  the  week,  approximately  70%  of 
average  daily  demand.  From  this  low  point,  demand  peaks  on 
Monday  and  Tuesday  at  approximately  113%  of  average  demand. 

From  Tuesday  through  Saturday,  demand  declines  almost  linearly 
(Van  Nuys  demand  exhibits  a sub-peak  on  Fridays) . Saturday 
demands  in  both  cases  are  approximately  90%  of  average  daily 
demand . 

In  Figures  3-6  and  3-7,  hourly  demand  patterns  for  all 
incoming  calls  and  for  Van  Nuys  trunk  calls  are  presented. 

These  patterns  were  derived  by  averaging  hourly  readings  for 
each  Monday  of  the  test  period.  Mondays  were  chosen  for  this 
hourly  analysis  since  they  are  generally  the  peak  demand  day  of 
the  week.  The  two  figures  show  similar  patterns  for  the  Van 
Nuys  trunk  and.  the  total  operation.  In  the  morning  hours, 
incoming  calls  build  from  a low  point  of  approximately  30%  of 
average  hourly  demand  at  6 am  to  peak  of  150%  of  average  hourly 
demand  between  10  and  11  am.  From  this  point  on,  demand  gra- 
dually tapers  off  to  nearly  the  average  hourly  rate  by  6 pm. 


3.3.2  Characteristics  of  System  Productivity 

Of  prime  concern  in  evaluation  of  CCIS  is  the  number  of 
calls  which  are  actually  answered  by  information  agents.  As 
noted  previously,  a substantial  proportion  of  callers,  especially 
during  peak  demand  periods,  hand  up  before  reaching  an  informa- 
tion agent.  Since  the  ACD  system  records  these  "lost"  calls, 
and  since  calls  answered  by  CCIS  agents  are  recorded  on  the 
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FIGURE  3-4.  TOTAL  CALLS  RECEIVED,  ALL  TRUNKS, 
BY  DAY  OF  WEEK  (6  am  - 5 pm) 

AVERAGE  FOR  ENTIRE  TEST  PERIOD 


Source:  ACD  Trunk  Group  Readings 
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FIGURE  3-5.  TOTAL  DAILY  CALLS  RECEIVED, 
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CALLS  RECEIVED 


FIGURE  3-6.  TOTAL  CALLS  RECEIVED,  ALL  TRUNKS, 
BY  HOUR  OF  DAY,  MONDAYS  ONLY 
AVERAGE  FOR  ENTIRE  TEST  PERIOD 


Source:  ACD  Trunk  Group  Readings 
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CCIS  log  file,  it  is  possible  to  make  an  important  comparison 
between  the  overall  productivities,  in  terms  of  calls  answered 
and  calls  lost,  of  manual  and  CCIS  agents. 

In  Figures  3-8  and  3-9,  average  daily  calls  answered  by 
CCIS  (Van  Nuys)  and  manual  (non-Van  Nuys)  agents  are  presented. 
The  points  plotted  in  these  graphs  were  derived  by  subtracting 
the  CCIS  log  file  average  daily  call  counts  from  the  average 
daily  calls  answered  for  the  Van  Nuys  trunk  and  the  entire 
Information  Section  for  each  week  of  the  test  period.  Figure 
3-8  shows  that  CCIS  agents  answered  an  average  of  570  calls 
daily  between  6 am  and  5 pm  during  the  test  period.  Figure  3-9 
shows  that  the  25  to  30  manual  agents  on  duty  from  6 am  to  5 pm 
each  day  answered  an  average  of  approximately  3550  calls. 

Figure  3-10  presents  a comparison  of  CCIS  versus  manual 
agent  productivity;  namely,  the  percentage  obtained  when  answered 
calls  (obtained  from  CCIS  log  file  call  counts)  for  each  group 
of  agents  are  divided  into  total  calls  received  (obtained  from 
the  ACD  Trunk  Group  Readings) . Average  daily  statistics  for 
each  week  of  the  test  period,  as  shown  in  Figures  3-2  and  3-9, 
were  used  in  this  calculation.  Over  the  test  period,  CCIS 
agents  answered  approximately  59%  of  all  Van  Nuys  calls.  Their 
trend,  as  shown  in  the  solid  line  Figure  3-10,  indicates  CCIS 
agents  were  generally  more  productive  over  the  first  six  weeks 
of  the  test  period,  during  which  time  the  average  daily  calls 
answered  approximated  63%  of  all  calls  received.  In  the  last  6 
weeks  of  the  test  period  for  which  log  file  data  exists  (as 
noted  earlier,  the  week  of  April  26  to  May  2 was  missing), 
calls  answered  by  CCIS  agents  approximated  only  55%  of  calls 
received.  The  maximum  observed  CCIS  productivity  occurred 
during  the  week  of  March  15  - 21,  in  which  66%  of  all  Van  Nuys 
calls  were  answered;  the  minimum  occurred  during  the  week  of 
May  10  - 16,  in  which  only  49%  of  Van  Nuys  calls  were  answered. 
The  dotted  line  in  Figure  3-10  presents  productivity  statistics 
derived  in  a similar  fashion  for  manual  (non-Van  Nuys)  agent 
productivity.  The  points  plotted  in  this  dotted  line  show  that 
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FIGURE  3-8.  DAILY  VAN  NUYS  TRUNK  CALLS  ANSWERED: 
AVERAGE  FOR  EACH  WEEK  OF  THE  TEST  PERIOD 


Source:  CCIS  Log  File 

* Missing  from  CCIS  Log  File 
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FIGURE  3-9.  TOTAL  DAILY  CALLS  ANSWERED  BY  MANUAL 
(NON -VAN  NUYS)  AGENTS 
AVERAGE  FOR  EACH  WEEK  OF  TEST  PERIOD 


Source:  ACD  Trunk  Group  Readings 

* Missing  from  CCIS  Log  File  Data 
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FIGURE  3-10.  DAILY  PERCENTAGE  OF  ALL  VAN  NUYS 
TRUNK  CALLS  ANSVJERED  BY  CCIS  AGENTS  (SOLID  LINE) 
and  non-van  NUYS  TRUNK  CALLS  ANSWERED  BY  MANUAL  AGENTS  (DOTTED 
LINE) : AVERAGE  FOR  EACH  WEEK  OF  THE  TEST  PERIOD 


Source:  ACD  Trunk  Group  Readings 


* Missing  from  CCIS  Log  File 
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manual  agent  productivity,  in  terms  of  calls  answered,  was 
quite  variable  from  week  to  week,  ranging  from  54%  of  all  calls 
answered  in  the  week  of  May  17-23,  to  almost  71%  of  such  calls 
in  the  week  of  May  10-16.  The  average  share  of  non-Van  Nuys 
calls  answered  by  manual  agents  over  the  entire  test  period 
was  approximately  61%. 

Comparison  of  the  solid  (CCIS)  and  dotted  (manual)  lines 
in  Figure  3-10  leads  to  two  interesting  conclusions.  The 
first  conclusion  is  the  rather  important  fact  that  implementa- 
tion of  the  CCIS  has  not,  based  on  these  data,  created  an 
appreciable  increase  over  manual  methods  in  the  overall  per- 
centage of  calls  answered  during  the  test  period.  The  CCIS 
agents  answered  a higher  percentage  of  calls  received  via  the 
Van  Nuys  trunk  than  did  manual  agents  on  their  respective 
trunks  for  5 of  the  first  6 weeks  in  the  test  period.  Con- 
versely, the  CCIS  agents  answered  a lesser  percentage  in  all 
but  the  final  of  the  last  7 weeks.  The  overall  percentages  of 
calls  answered  by  both  groups  over  the  entire  test  period, 
however,  are  quite  comparable:  59%  for  the  CCIS  agents  versus 
61%  for  the  manual  agents. 

The  second  conclusion  that  can  be  drawn  from  comparing 
the  two  agent  productivity  lines  on  Figure  3-10  is  that  CCIS 
productivity  in  terms  of  daily  answered  calls  for  some  reason 
exhibits  an  inverse  relationship  to  that  for  manual  agents. 

In  9 out  of  the  12  weeks  for  which  this  comparison  was  possible, 
CCIS  agent  productivity  increased  while  manual  agent  productivity 
decreased,  or  vice  versa.  During  the  weeks  of  April  5-11  and 
April  19-25,  productivity  for  both  groups  increased,  while 
they  both  declined  during  the  week  of  March  29  - April  4. 

Several  caveats  to  the  above  productivity  comparison 
between  CCIS  and  manual  agents  should  be  considered.  The 
first  caveat  is  that  caller  demands  for  both  CCIS  and  manual 
services  originated  from  different  locations  in  a large  metro- 
politan area,  and  these  demands  (type  and  length  of  caller 
queries,  for  example)  might  be  quite  different,  requiring 
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different  types  of  agent  responses  and  taking  different  lengths 
of  tiine. 

A second  caveat  is  that  Information  Section  staffing,  in 
terms  of  agents  on  duty  per  call  received,  varies  throughout 
the  course  of  the  day,  and  may  well  have  been  quite  different 
for  CCIS  and  manual  agents  for  some  or  all  of  the  test  period 
in  terms  of  number  of  available  agents  per  hourly  caller  demand. 
Since  daily  staffing  data  were  not  available,  derivation  of 
true  productivity  comparisons  between  manual  and  CCIS  agents 
(in  terms  of  calls  answered  per  man-hour  worked)  were  not 
possible . 

A third  caveat,  as  explained  above,  is  that  CCIS  agents 
were  occasionally  confronted  with  situations  in  which  they  were 
forced  to  re  sort  to  manual  data  retrieval.  Thus,  no  producti- 
vity comparison  between  the  two  groups  is  completely  unbiased. 

A final  caveat  to  the  above  comparison  is  that  the  number 
of  lost  calls  for  the  CCIS  and  manual  groups  might  be  a function 
of  variables  other  than  agent  productivity.  For  example, 
callers  averse  to  waiting  "on  hold"  for  an  agent  might  hang  up 
immediately  after  hearing  the  recorded  message  of  the  ACD  system, 
even  in  situations  where  productivity  was  quite  high  and  the 
caller  holding  period  was  very  brief.  Thus,  basing  agent  pro- 
ductivity comparisons  completely  on  the  overall  percentage  of 
calls  answered  might  not  be  totally  valid. 

3.3.3  Detailed  Characteristics  of  CCIS  Calls 

In  Figures  3-11  through  3-17,  a variety  of  detailed  data 
concerning  CCIS  agent  performance  as  compiled  by  the  CCIS  log 
file  are  presented.  As  noted  previously,  these  data  apply  only 
to  those  calls  answered  by  CCIS  agents  using  CCIS  equipment. 

Figure  3-11  displays  the  average  duration  of  each  CCIS  call 
for  each  week  of  the  test  period.  These  call  times,  together 
with  their  respective  standard  deviations,  are  also  presented 
in  Table  3-2.  It  is  clearly  evident  from  these  data  that  there 
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FIGURE  3-11 


. AVERAGE  CALL  TIMES,  CCIS  CALLS  ANSWERED 
FOR  EACH  WEEK  OF  THE  TEST  PERIOD 


SOURCE:  CCIS  Log  File 

* Missing  from  CCIS  Log  File 
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5/24-5/30 


TABLE  3-2.  AVERAGE  CALL  TIMES  AND  STANDARD  DEVIATIONS 
FOR  EACH  WEEK  OF  THE  TEST  PERIOD 


I • 


WEEK 

AVERAGE 
CALL  TIME 

STANDARD 

DEVIATION 

2/22  - 2/28 

2:09 

1 : 26 

3/1  - 3/7 

2:05 

1 : 23 

3/8  - 3/14 

2:02 

1 : 24 

3/15  - 3/21 

2:04 

1 : 21 

3/22  - 3/28 

2:01 

1 : 22 

3/29  - 4/4 

2:02 

1 : 22 

4/5  - 4/11 

1 : 55 

1 : 16 

4/12  - 4/18 

2:03 

1 : 25 

4/19  - 4/25 

2:11 

1:31 

4/26  - 5/2* 

* 

★ 

5/3  - 5/9 

1 : 55 

1 : 20 

5/10  - 5/16 

1:56 

1 : 28 

5/17  - 5/23 

1:56 

1:18 

5/23  - 5/30 

1 : 53 

1:15 

* Missing  from  CCIS  Log  File 
Source:  CCIS  Log  File 
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was  a pronounced  downward  trend  in  CCIS  call  times  over  the 
course  of  the  test  period,  with  the  exception  of  a two-week 
period  in  April  in  which  call  times  and  standard  deviations 
both  rose  substantially  (April  12-25).  Call  times  ranged  from 
a high  of  2:11  during  the  week  of  April  19-25  to  a low  of  1:53 
during  the  week  of  May  24-30.  The  average  call  time  for  the 
entire  test  period  was  2:01. 

The  statistics  presented  in  Figure  3-12  are  those  derived 
when  the  call  times  of  Figure  3-11  are  divided  into  3600,  the 
number  of  seconds  per  hour.  The  result  is  the  average  "projected" 
number  of  CCIS  calls  per  hour  answered  for  each  week  of  the 
test  period.  The  hourly  call  count  is  the  SCRTD ' s chief  measure 
of  agent  productivity.  SCRTD  agent  training  focuses  on  tech- 
niques for  increasing  hourly  call  counts;  agents  at  the  end  of 
their  training  period  are  expected  to  answer  at  least  20  calls 
per  hour.  The  "projected"  call  count,  of  course,  rather 
unrealistically  assumes  no  agent  "break"  time  between  calls. 

As  can  be  seen  from  Figure  3-12,  CCIS  projected  hourly  call 
counts  exceed  the  SCRTD  standard  substantially,  even  for  those 
weeks  in  which  call  times  were  longest.  The  average  projected 
hourly  call  count  for  the  entire  test  period  was  approximately 
29.9,  49.3%  higher  than  the  SCRTD  standard. 

By  pressing  a "release"  button  on  the  telephone  console,  it 
is  possible  for  an  agent  to  interrupt  the  routing  of  the  next 
call  to  his  or  her  work  station,  thus  allowing  a break  from  the 
routine.  The  large  discrepancy  between  actual  call  counts  (20 
per  hour)  and  "projected"  call  counts  (30  per  hour)  suggests 
that  agents  might  have  been  abusing  this  release  mechanism 
during  the  system  evaluation  period.  In  late  1981,  when  new 
telephone  equipment  was  installed,  SCRTD  management  was  able 
for  the  first  time  to  monitor  and  control  the  extent  to  which 
each  agent  used  the  release.  A maximum  of  12  minutes  of  break 
time  per  8-hour  shift  was  established,  and  average  agent  call 
counts  increased  considerably  to  approximately  25  per  hour. 
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FIGURE  3-12.  PROJECTED  CCIS  CALLS  PER  HOUR  PER  AGENT 
AVERAGE  FOR  EACH  WEEK  OF  THE  TEST  PERIOD 


SOURCE:  CCIS  Log  File 

* Missing  from  CCIS  Log  File 
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Figure  3-13  presents  a variable  of  considerable  interest  to 
the  evaluation.  The  percentage  of  CCIS  calls  (averaged  for 
each  week  of  the  test  period)  which  were  answered  "manually;" 
that  is,  those  for  which  no  CCIS  transactions  were  undertaken. 

Such  calls  could  be  of  two  different  types.  They  could  be 
those  in  which  the  customer's  origin  or  destination  fell  outside 
the  CCIS  geographic  data  base,  or  those  in  which  the  agent  knew 
the  answer  to  the  caller's  query  without  recourse  to  any  data 
sources.  The  first  type  of  "manual"  call  would  require  an 
agent  to  retrieve  data  from  printed  materials,  and  would  pre- 
sumably take  longer  than  the  average  CCIS-assi sted  call.  The 
second  type  would  require  no  retrieval,  and  would  presumably  be 
shorter  than  the  average  CCIS  call.  During  the  test  period,  it 
is  evident  from  data  collected  in  the  CCIS  log  file  that  the 
preponderance  of  these  "manual"  calls  were  of  the  second  type, 
as  average  manual  call  times  reported  are  almost  invariably 
faster  than  CCIS  call  times.  On  average  over  the  entire  test 
period,  they  average  approximately  25  seconds  faster  than  CCIS 
calls.  The  percentage  of  these  manual  calls  on  the  CCIS  remained 
fairly  constant  over  the  test  period,  averaging  about  24%  of 
all  CCIS  calls  answered.  The  highest  percentage  occurred  during 
the  first  week  of  the  test  period,  February  22-28,  when  29.6% 
of  all  calls  were  answered  manually.  The  lowest  observed  per- 
centage occurred  the  following  week,  March  1-7,  when  19.9%  of 
all  calls  were  handled  manually. 

Table  3-3  and  Figure  3-14  present  a breakdown  of  all  CCIS 
calls  classified  by  duration  of  call,  as  averaged  for  each  week 
of  the  test  period.  These  data  show  that  the  percentage  of 
calls  answered  in  less  than  one  minute  remained  relatively 
constant,  increasing  slightly  towards  the  end  of  the  test  period. 
The  percentage  of  calls  answered  in  less  than  1 minute  during 
the  test  period  ranged  from  18.4  in  the  week  of  March  1-7  to 
25.3  in  the  week  of  May  24-30,  with  an  average  value  of  20.0. 
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FIGURE  3-13.  PERCENT  OF  CCIS  CALLS  ANSWERED  MANUALLY 
(NO  SYSTEM  TRANSACTIONS) 

AVERAGE  FOR  EACH  WEEK  OF  THE  TEST  PERIOD 


Source:  CCIS  Log  File 

* Missing  from  CCIS  Log  File  Data 
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TABLE  3-3.  CLASSIFICATION  OF  CCIS  CALLS 
ANSWERED  BY  DURATION  OF  CALL 


WEEK 

LESS  THAN 
ONE 

MINUTE 

LESS  THAN 
TWO 

MINUTES 

LESS  THAN 
THREE 
MINUTES 

OVER 

THREE 

MINUTES 

2/22-2/28 

21.6 

57.9 

78.5 

21.5 

3/1-3/7 

18.4 

59.3 

79.9 

20.1 

3/8-3/14 

21 . 3 

60.4 

81.4 

18.6 

3/15-3/21 

21 . 5 

58.1 

79.7 

20.3 

3/22-3/28 

20.9 

60.2 

82.9 

18.1 

3/29-4/4 

21 . 3 

60.3 

79 . 8 

20.2 

4/5-4/11 

23.5 

62.7 

84.0 

16.0 

4/12-4/18 

21 . 5 

60.8 

81.0 

19.0 

4/19-4/25 

18.9 

57.4 

78.4 

21.6  1 

4/26-5/2* 

* 

★ 

* 

* 1 

1 

5/3-5Z9 

24.7 

63.9 

83.9 

16.1 

5/10-5/16 

24.2 

63.2 

84.1 

16.0 

5/17-5/23 

23.4 

62.5 

83.0 

17.0 

5/24-5/30 

25.3 

64.5 

83.8 

16.2 

AVERAGE 

22.0 

60.9 

81.5 

18.5 

* Missing  from  CCIS  Log  File  Data 
Source:  CCIS  Log  File 
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FIGURE  3-14.  PERCENTAGES  OF  CCIS  CALLS  ANSWERED 
HAVING  DURATIONS  OF  ONE,  TWO,  OR  THREE  MINUTES  OR  LESS 
AVERAGES  FOR  EACH  WEEK  OF  THE  TEST  PERIOD 


Source:  CCIS  Log  File 

* Missing  from  CCIS  Log  File  Data 
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Calls  answered  in  two  minutes  or  less  show  a gradual  upward 
trend  through  the  test  period  with  the  exception  of  the  two- 
week  period  from  April  11-25.  The  percentage  of  2-minute 
duration  calls  ranged  from  57.4  in  the  week  of  April  19-25  to 
64.5  in  the  week  of  May  24-30,  with  an  average  value  for  the 
test  period  of  60.9.  Calls  answered  in  3 minutes  or  less  (the 
SCRTD  standard  of  20  calls  per  hour)  also  exhibit  an  upward 
trend  throughout  the  test  period,  ranging  from  78.4  percent  of 
all  calls  during  the  week  of  April  19-25  to  84.1  percent  of  all 
calls  during  the  week  of  May  10-16.  The  average  percentage  of 
calls  answered  in  3 minutes  or  less  for  the  entire  test  period 
was  81.5.  Conversely,  the  average  percentage  of  calls  taking 
longer  than  3 minutes  to  complete  was  18.5. 

Figure  3-15  presents  a plot  of  CCIS  user  errors  per  call 
answered  averaged  for  each  week  over  the  test  period.  The 
CCIS  actually  kept  track  of  three  different  types  of  errors: 

CCIS  internal  (software)  errors,  input-output  (hardware) 
errors,  and  user  errors.  Errors  of  the  first  two  types  were 
extremely  infrequent,  occurring  on  less  than  five  occasions 
during  the  entire  test  period.  User  errors  which  include 
typing,  spelling  and  other  data  entry  errors  were  more  frequent, 
ranging  from  one  error  in  every  3.57  calls  in  the  first  week 
of  the  test  period  (February  22-28)  to  one  error  in  every 
6.67  calls  in  the  final  week  (May  24-30).  Over  the  entire 
period,  CCIS  agents  averaged  one  error  every  5 calls.  The 
points  plotted  in  Figure  3-15  show  a generally  decreasing 
trend  in  user  errors  per  call  with  the  exception  of  the  two- 
week  period  from  April  11-25. 

The  CCIS  was  able  to  measure  and  record  times  for  various 
segments  of  each  CCIS  call.  Calls  can  be  divided  into  three 
separate  components:  the  caller  query,  the  machine  interaction 
(agent  keyboard  input,  machine  transaction  processing  and 
response  screen  display) , and  agent  response.  Average  durations 
for  each  of, these  call  segments  for  each  week  of  the  test 
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WEEK 


FIGURE  3-15.  USER  ERRORS  DETECTED  BY  CCIS 
PER  CALL  ANSWERED 

AVERAGE  FOR  EACH  WEEK  OF  TEST  PERIOD 


Source:  CCIS  Log  File 

* Missing  from  CCIS  Log  File  Data 
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period  are  presented  in  Figure  3-16,  and  tabulated  in  Table 
3-4.  From  these  data,  it  can  be  seen  that  machine  interaction 
and  agent  response  times  remained  quite  stable  over  the  course 
of  the  test  period,  averaging  24  and  49  seconds,  respectively. 
Only  during  the  two-week  period  of  April  12-25  were  three 
major  fluctuations  in  durations  for  these  two  call  segments. 
Average  durations  of  caller  queries,  on  the  other  hand,  show 
a pronounced  downward  trend  over  the  test  period,  with  the 
exception  of  April  19-25. 

Figure  3-17  portrays  a further  breakdown  of  the  machine 
interaction  call  segment  shown  in  Figure  3-16.  This  segment 
can  be  divided  into  two  parts:  first,  the  amount  of  time 
spent  by  the  agent  entering  appropriate  information  on  the 
terminal  keyboard,  and  second,  the  amount  of  time  spent  by 
the  computer  in  retrieving  and  displaying  the  appropriate 
response.  As  can  be  seen  in  Figure  3-17,  actual  retrieval 
time  takes  a small  fraction  of  total  machine  interaction  time 
(an  average  of  14%  for  the  entire  test  period).  Retrieval 
took  an  average  of  only  3%  of  total  call  time.  Over  the  test 
period,  96.3%  of  all  machine  responses  were  given  in  7 seconds 
or  less,  the  performance  standard  originally  specified  by  the 
SCRTD. 

3.4  CGIS  CONTROLLED  EXPERIMENT  RESULTS 

The  CGIS  controlled  experiment  conducted  by  the  evaluation 
contractor  produced  separate  call  times  for  each  test  agent 
and  each  test  question.  For  all  9 test  agents,  the  average 
call  time  for  all  test  agents  was  slightly  over  four  minutes 
per  call.  Table  3-5  presents  average  agent  call  times  for 
each  of  the  4 nine-question  sections  of  the  test  broken  down 
by  agent  skill/experience  level.  The  test  was  broken  down 
into  4 sections  in  order  to  reduce  the  number  of  participants 
(and  tests)  required  for  analysis  of  variance  results  to  be 
obtained.  As  might  be  expected,  the  novice  agents  took  longer 
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CALL  SEGMENT  DURATION,  SECONDS 
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FIGURE  3-16.  CCIS  CALL  SEGMENT  DURATIONS 
AVERAGE  FOR  EACH  WEEK  OF  THE  TEST  PERIOD 


Source:  CCIS  Log  File 

* Missing  from  CCIS  Log  File  Data 
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TABLE  3-4.  CCIS  CALL  SEGMENT  DURATIONS 
AVERAGES  FOR  EACH  WEEK  OF  THE  TEST  PERIOD 


WEEK 

CALLER 

QUERY 

MACHINE 

INTERACTION 

AGENT 

RESPONSE 

TOTAL 
CALL  TIME 

2/22-2/28 

: 54 

:25 

:48 

2:07 

3/1-3/7 

:51 

: 26 

:48 

2:05 

3/G-3/14 

; 50 

:25 

:47 

2:02 

3/15-3/21 

: 51 

:26 

:47 

2:04 

3/22-3/28 

:48 

: 24 

:49 

2:01 

3/29-4/4 

:51 

: 22 

:49 

2:02 

4/5-4/11 

;47 

:19 

:49 

1 : 53 

4/12-4/18 

: 47 

: 27 

:49 

2:03 

4/19-4/25 

: 58 

:19 

:54 

2:11 

4/26-5/2* 

* 

* 

* 

;*r 

5/3-5/9 

: 45 

: 22 

:48 

1 : 55 

5/10-5/16 

:42 

:25 

:49 

1 : 56 

5/17-5/23 

:43 

; 25 

:48 

1:56 

5/24-5/30 

:44 

:22 

:47 

1 : 53 

AVERAGE 

: 49 

: 24 

:49 

2:01 

* Missing  from  CCIS  Log  File  Data 
Source:  CCIS  Log  File 
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PERCENT  OF  TOTAL  CALL  TIME 
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FIGURE  3-17.  MACHINE  INTERACTION  AND  DATA  RETRIEVAL 
TIMES  AS  PERCENTAGES  OF  TOTAL  CALLS  TIMES 
AVERAGES  FOR  EACH  WEEK  OF  THE  TEST  PERIOD 


Source:  CCIS  Log  File  data 

* Missing  from  CCIS  Log  File  Data 
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on  average  to  complete  their  calls  than  did  the  more  experienced 
agents.  Call  times  for  the  novice  agents  as  a group  averaged 
over  five  minutes  while  call  times  for  intermediate  and  advanced 
agents  averaged  approximately  three  and  one-half  minutes. 

Call  time  differences  between  intermediate  and  advanced  agents 
were  less  dramatic,  with  the  intermediate  agents  actually 
performing  slightly  faster  (seven  seconds  on  average)  than 
the  advanced  agents  on  all  of  the  nine-question  test  sections. 
These  results  might  suggest  that  average  call  times  decrease 
with  increasing  job  experience  only  up  to  a point,  beyond 
which  call  times  remain  relatively  stable. 

Table  3-6  presents  average  agent  call  times  broken  down 
by  mode  of  data  retrieval.  The  agents  using  the  CCIS  mode  of 
data  retrieval  had  the  fastest  overall  call  times,  averaging 
approximately  three  minutes  and  51  seconds  per  response.  The 
manual  mode  agents  ranked  second  overall  with  an  average  call 
time  of  four  monutes  and  three  seconds,  while  the  mixed  mode 
agents  were  third  overall  with  an  average  call  time  of  four 
minutes  and  twenty  seconds.  Analysis  of  variance  results 
show  that  there  were  few  statistically  significant  differences 
between  modes;  they  do  show,  however,  that  the  order  in  which 
test  questions  were  asked  did  not  affect  agent  responses. 

Table  3-7  provides  a comparison  of  individual  agent  perfor- 
mance in  the  experiment  as  measured  by  projected  hourly  call 
rates,  i.e.  average  call  times  divided  into  3600  seconds  per 
hour.  As  can  be  seen  from  this  table,  the  fastest  overall 
were  the  intermediate  and  advanced  CCIS  agents,  followed  by 
the  intermediate  mixed  and  manual  mode  agents.  The  advanced 
mixed  and  manual  agents  are  next,  followed  by  the  novice 
agents  in  the  manual,  CCIS,  and  mixed  modes,  respectively. 

The  slowest  agents,  by  far,  were  the  novice  CCIS  and  the 
novice  mixed  mode  agent,  who  used  CCIS  most  of  the  time. 
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TABLE  3-5.  AVERAGE  AGENT  CALL  TIMES 
GROUPED  BY  EXPERIENCE/SKILL  LEVEL 
(IN  MINUTES  AND  SECONDS) 


LEVEL  OF 
AGENT 
SKILL/ 
EXPERIENCE 

QUESTION  SET 
(Question  Numbers) 

I 

(1-9) 

II 

(10-18) 

III 

(19-27) 

IV 

(28-36) 

TOTAL 

(1-36) 

NOVICE 

6 : 02 

5:50 

4:14 

4 : 52 

5:15 

INTERMEDIATE 

3 : 48 

3 : 32 

2:58 

3:27 

3 : 27 

ADVANCED 

4:11 

3 : 56 

2:41 

3 : 28 

3 : 34 

ALL  AGENTS 

4:40 

4 : 26 

3 : 18 

3 : 55 

4:05 

Source:  CGIS  Controlled  Experiment 


(Note:  The  experiment  was 
questions  each  in  order  to 
performance  of  analysis  of 
squares;  this  also  limited 
nine  . ) 


divided  into  four  sets  of  nine 
obtain  information  suitable  for 
variance  in  9x9  element  latin 
the  number  of  test  subjects  to 
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TABLE  3-6.  AVERAGE  AGENT  CALL  TIMES 
GROUPED  BY  MODE  OF  DATA  RETRIEVAL 
(IN  MINUTES  AND  SECONDS) 


MODE  OF 
DATA 

RETRIEVAL 

QUESTION  SET 
(Question  Numbers) 

I 

(1-9) 

II 

(10-18) 

III 

(19-27) 

IV 

(28-36) 

TOTAL 

(1-36) 

MANUAL 

5:01 

4:08 

3:28 

3:35 

4:03 

CCIS 

3 : 58 

4 : 30 

3:07 

3 : 50 

3:51 

MIXED 

5 : 02 

4 : 40 

3:18 

4 : 21 

4 : 20 

ALL  AGENTS 

4 : 40 

4 : 26 

3:18 

3:55 

4:05 

Source:  CCIS  Controlled  Experiment 
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TABLE  3-7.  AVERAGE  CALL  RATES  FOR  EACH  TEST  AGENT 

(IN  CALLS  PER  HOUR) 


AGENT 

SKILL/ 

EXPERIENCE 

LEVEL 

MODE  OF  DATA  RETRIEVAL 

MANUAL 

CCIS 

MIXED 

NOVICE 

13.1 

11.1 

10.4 

INTERMEDIATE 

16.3 

19.1 

17.1 

ADVANCED 

15.3 

19.9 

15.9 

Source:  CCIS  Controlled  Experiment 
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Another  variable  of  interest  in  comparing  agent  per- 
formance is  the  variability  in  the  call  times  recorded.  When 
the  standard  deviations  of  agent  call  times  were  calculated 
they  showed  that  the  intermediate  and  advanced  CCIS  agents 
were  the  most  consistent  in  their  call  times,  while  the  novice 
manual  and  mixed  mode  agents  were  the  least  consistent. 

The  data  that  were  presented  in  Tables  3-5,  3-6,  and  3-7 
clearly  show  that  the  nine  agents  participating  in  the  con- 
trolled experiment  posted  average  projected  hourly  call  rates 
that  were  generally  below  the  SCRTD  standard  of  20  calls  per 
hour,  and  far  below  the  average  projected  call  rates  calculated 
by  the  CCIS  log  file  during  the  test  period.  Three  different 
explanations  might  be  advanced  for  this  discrepancy  between 
experimental  and  "real  world"  results.  They  are: 

• Experimental  effects; 

• Difficulty  of  test  questions;  and 

• Agent  experience. 

These  are  discussed  below: 

Experimental  effects  are  those  factors  influencing  agent 
performance  arising  out  of  the  experiment  itself.  Because  of 
the  configuration  of  the  telephone  system  at  SCRTD,  it  was 
impossible  to  test  the  agents  at  their  normal  work  stations. 
Instead,  they  were  isolated  in  one  of  their  training  rooms 
for  the  experiment,  so  that  the  calls  could  be  placed  to  them 
directly.  Because  of  this  isolation,  agents  participating  in 
the  experiment  may  have  felt  for  some  reason  they  were  being 
singled  out  for  closer  observation.  Furthermore,  because 
they  were  advised  that  call  time  and  response  accuracy  were 
being  measured,  they  may  have  felt,  despite  consultant 
reassurances  to  the  contrary,  that  the  experiment  was  designed 
to  give  SCRTD  management  an  in-depth  view  of  their  job  performance. 
For  these  reasons,  agents  may  well  have  been  both  more  nervous 
and  more  cautious  in  their  responses  than  they  would  ordinarily 
have  been  at  their  own  work  stations.  Both  of  these  factors 
would  mitigate  towards  longer  responses. 
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Agents  and  SCRTD  management  both  agreed  that  the  queries 
selected  for  use  in  the  experiment  were  generally  more  complex 
than  average  calls  encountered  by  agents  during  the  normal 
performance  of  their  jobs.  This  is  not  surprising,  since 
test  questions  were  constructed  to  measure  the  itinerary- 
building capabilities  of  the  CCIS  as  well  as  the  manual 
agents.  As  shown  in  the  CCIS  log  file  data,  experienced 
agents  are  able  to  respond  to  certain  familiar  and  frequently- 
recurring  queries  without  reference  to  any  data  about  25%  of 
the  time.  These  "off  the  top  of  the  head"  responses  obviously 
take  far  less  time  than  those  for  which  either  manual  or 
computer  data  retrieval  is  required,  and  tend  to  increase 
substantially  an  agent's  average  call  rate.  None  of  the 
queries  in  the  experiment  was  so  easy  that  data  retrieval  was 
unnecessary.  In  fact,  all  of  the  queries  requested  construction 
of  a complete  itinerary  from  point  A to  point  B,  and  most 
involved  further  information  references  beyond  this  itinerary 
request.  As  a result,  the  difficulty  of  the  test  questions 
may  well  have  been  a factor  in  producing  longer  agent  call 
times. 

The  novice  agents  ranked  seventh,  eighth,  and  ninth  in 
terms  of  overall  call  productivity.  Their  relatively  slow 
performance  tended  to  diminish  the  overall  call  rates  for 
each  retrieval  mode.  This  effect  was  especially  true  for  the 
CCIS  mode,  where  the  average  call  rate  for  the  novice  was  a 
full  eight  calls  per  hour  slower  than  that  of  the  advanced 
agents. 

For  a more  detailed  presentation  of  CCIS  Controlled 
Experiment  methodology  and  results,  see  the  companion  report 
entitled  The  CCIS  Experiment:  Comparing  Data  Retrieval  Modes 
at  the  SCRTD. 
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3.5  QUALITATIVE  DATA  COLLECTION 

The  wealth  of  quantitative  data  collected  by  the  system 
evaluation  contractor  and  the  SCRTD  yielded  the  very  detailed 
descriptions  of  caller  demand,  agent  performance,  and  call 
characteristics  described  above.  They  did  not,  however, 
provide  enough  information  to  satisfy  all  the  objectives  of 
the  system  evaluation.  Many  aspects  of  CCIS  operations 
were  difficult  to  quantify.  For  this  reason,  a number  of 
qualitative  data  collection  activities  were  also  undertaken. 
These  qualitative  evaluation  activities  had  three  major 
objectives: 

• To  gain  insights  into  the  opinions  and  attitudes  of 
both  CCIS  and  manual  agents  concerning  the  system; 
its  overall  accuracy,  its  impact  on  their  normal  job 
functions,  its  influence  on  their  overall  job  satis- 
faction , etc. ; 

• To  ascertain  SCRTD  management  perceptions  as  to  how 
system  implementation  was  progressing,  how  effectively 
it  was  meeting  desired  objectives,  etc.;  and 

• To  obtain  suggestions  for  potential  improvements  or 
enhancements  to  the  system. 

Three  separate  techniques  were  employed  by  the  evaluation 
contractor  in  obtaining  this  qualitative  information.  They 
were ; 

• Survey  questionnaires  of  both  manual  and  CCIS  agents; 

• In-depth  interviews  of  key  SCRTD  personnel  involved 
in  system  implementation;  and 

• Qualitative  grading  of  agent  responses  obtained  in  the 
CCIS  controlled  experiment. 
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3.6  QUALITATIVE  EVALUATION  METHODOLOGY 


In  reviewing  alternative  methods  for  performing  an  evalua- 
tion of  the  qualitative  aspects  of  the  CCIS,  the  method  which 
appeared  to  provide  the  most  complete  understanding  of  the 
system's  impacts  on  agent  and  supervisor  attitudes  was  a 
before-and-af ter  approach.  Based  on  this  decision,  two  separate 
sets  of  agent  attitudinal  surveys  were  designed  and  administered, 
and  two  separate  series  of  management  interviews  were  conducted, 
before  and  after  the  90-day  quantitative  evaluation  period. 

The  "before"  surveys  and  interviews  were  conducted  in  October 
and  November  1980,  and  the  "after"  data  were  collected  in  June 
1981.  Because  it  was  considered  useful  to  determine  opinions 
of  both  manual  and  CCIS  agents,  a separate  survey  was  administered 
to  each  group  on  each  occasion. 

Qualitative  data  concerning  CCIS  operation  were  also  produced 
as  a result  of  the  CCIS  controlled  experiment.  Agent  responses 
to  each  of  the  36  test  questions  as  recorded  by  the  contractor 
in  the  experiment  were  graded  by  a panel  of  impartial  graders 
in  the  SCRTD  Planning  department  as  well  as  the  Information 
Section  Training  Supervisor.  In  the  grading  process,  major 
emphasis  was  placed  on  the  accuracy  and  completeness  of  each 
response.  Before  the  grading  process  began,  agent  identities 
and  their  modes  of  information  retrieval  were  removed  from 
response  logs  so  as  to  eliminate  any  possible  bias  on  grading 
results  induced  by  these  two  factors.  While  this  grading  pro- 
cess relied  on  subjective  judgments  and  consistency  of  the 
graders,  it  nevertheless  provided  a good  basis  for  distinguishing 
the  relative  quality  of  agent  responses. 


3.7  QUALITATIVE  EVALUATION  RESULTS 

In  the  following  section,  results  of  the  various  quali- 
tative data  collection  instruments  are  described.  The  "before" 
agent  survey  results  are  described  first.  Next,  the  "after" 
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survey  results  are  presented  and  compared  with  the  "before" 
results  where  possible.  Analyses  of  the  response  quality  grading 
of  the  CCIS  controlled  experiment  are  then  described.  Finally, 
summaries  of  interviews  of  eight  key  SCRTD  personnel  involved 
in  CCIS  implementation  are  provided. 


3.7.1  "Before"  Agent  Survey  Results 

The  "before"  surveys  for  manual  and  CCIS  agents  were  designed 
to  encourage  maximum  agent  participation.  Overall  length  was 
limited  to  approximately  15  questions,  most  of  which  were  pre- 
sented in  a multiple-choice  format.  For  several  questions, 
however,  blanks  were  provided  for  "open-ended"  responses  by 
agents  who  chose  to  volunteer  additional  information.  Response 
to  the  survey  was  excellent.  A total  of  25  CCIS  and  47  manual 
agent  surveys  were  completed  and  returned  for  processing.  Sur- 
vey responses  were  coded  numerically  by  the  consultant  and 
analyzed  using  a Statistical  Package  for  the  Social  Sciences 
(SPSS)  program  which  yielded  frequencies  for  each  possible 
response  type  as  well  as  cross- tabl nations  of  certain  data 
items  of  interest. 

3. 7. 1.1  Manual  Agent  "Before"  Survey  Results  - Manual  agents 
were  asked  a total  of  12  questions  of  which  7 allowed  multiple- 
choice  response  selection  and  5 provided  for  open-ended  responses. 
Agent  responses  to  each  are  presented  below: 

• The  first  questions  in  each  of  the  surveys  administered 
dealt  with  agent  experience  and  call  count  under  busy 
conditions,  i.e.,  conditions  under  which  the  agent  can 
answer  calls  continuously.  The  first  columns  of  Tables 
3-8  and  3-9  present  the  manual  agent  "before"  responses 
to  these  questions.  Manual  agents  responding  to  the 
survey  tended  to  be  well-experienced,  with  an  average 
job  tenure  of  at  least  22  months.  Their  self- admitted 
call  counts  generally  ranged  from  the  upper  teens  to 
low  twenties  per  hour. 


-83- 


TABLE  3-8.  RESPONSES  TO  SURVEY  QUESTIONS 
REGARDING  AGENTS'  EXPERIENCE 
(%  OF  TOTAL  AGENTS  RESPONDING) 


AGENT 

EXPERIENCE 

MANUAL 

CGIS 

BEFORE 

SURVEY 

AFTER 

SURVEY 

BEFORE 

SURVEY 

AFTER 

SURVEY 

0 - 6 Months 

9 

7 

4 

0 

7-12  Months 

6 

3 

0 

0 

1-2  Years 

38 

23 

32 

23 

2-3  Years 

6 

17 

16 

29 

Over  3 Years 

40 

50 

48 

48 

TOTAL  AGENTS 
RESPONDING 

47 

30 

25 

31 
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TABLE  3-9.  RESPONSES  TO  SURVEY  QUESTIONS 
REGARDING  AGENTS'  HOURLY  CALL  COUNTS 
(%  OF  TOTAL  AGENTS  RESPONDING) 


HOURLY 

MANUAL 

CCIS 

CALL  COUNT 
UNDER  BUSY 
CONDITIONS 

BEFORE 

SURVEY 

AFTER 

SURVEY 

BEFORE 

SURVEY 

AFTER 

SURVEY 

15  or  Less 

4 

3 

0 

3 

16  - 20 

26 

30 

36 

23 

21  - 25 

49 

43 

52 

58 

26  - 30 

17 

20 

8 

13 

Over  30 

4 

3 

4 

3 

TOTAL 

AGENTS 

RESPONDING 

47 

30 

25 

31 

Each  survey  also  contained  questions  in  which  agents 
were  asked  to  rate  their  individual  job  performance 
and  job  satisfaction.  The  vast  majority  of  manual 
agents  in  the  "before"  survey  rated  their  abilities  as 
"excellent"  or  "good".  Manual  agent  job  satisfaction 
was  for  the  most  part  rated  "good",  although  10  agents 
responded  "fair"  or  "poor".  The  first  columns  of 
Table  3-10  and  3-11  present  these  self-ratings. 

Manual  agents  were  fairly  evenly  divided  on  the 
question  "Do  you  feel  that  you  have  reached  your  full 
potential  in  performing  as  a transit  information 
agent?".  A total  of  45%  replied  "yes"  to  this  question, 
while  38%  replied  "no";  and  17%  chose  the  "don't  know" 
response . 

Manual  agents  were  also  asked  the  frequency  with  which 
they  used  map  books  (Thomas  Guides)  in  providing  caller 
responses.  A similar  question  asked  the  frequency 
with  which  agents  used  no  references  whatever  in 
answering  calls.  These  frequencies  were  of  interest 
because  map  book  use  tends  to  slow  down  agent  call 
counts,  while  responses  "off  the  top  of  the  agent's 
head"  tend  to  increase  them.  The  responses  to  these 
two  questions  are  shown  in  Table  3-12.  They  show  that 
manual  agents  responding  to  the  survey  use  map  books 
quite  frequently,  and  no  references  somewhat  less  so. 

To  the  question  "What  part  of  your  job  causes  you  the 
most  problem?",  several  alternative  answers  were  pro- 
vided as  well  as  a blank  labelled  "other"  for  additional 
information  to  be  supplied  by  the  agent.  "Poor  com- 
prehension by  the  caller"  was  by  far  the  most  popular 
reply;  80%  of  the  manual  agents  chose  this.  This  was 
by  far  the  highest  total  for  any  single  response.  Only 
two  agents  chose  the  response  "understanding  the  caller's 
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TABLE  3-10.  RESPONSES  TO  SURVEY  QUESTIONS 
REGARDING  AGENTS'  ABILITY 
{%  OF  TOTAL  AGENTS  RESPONDING) 


MANUAL 

CCIS 

AGENT 

SELF-RATING 

CATEGORY 

BEFORE 

SURVEY 

AFTER 

SURVEY 

BEFORE 

SURVEY 

AFTER  SURVEY 

ABILITY 

USING 

CCIS 

ABILITY 

USING 

CCIS 

ABILITY 

USING 

MANUAL 

EXCELLENT 

38 

53 

16 

23 

52 

GOOD 

60 

43 

36 

58 

45 

FAIR 

2 

3 

32 

16 

3 

POOR 

0 

0 

16 

3 

0 

TOTAL 

AGENTS 

RESPONDING 

47 

30 

25 

31 

31 
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TABLE  3-11.  RESPONSES  TO  SURVEY  QUESTIONS 
REGARDING  AGENTS'  JOB  SATISFACTION 
(%  OF  TOTAL  AGENTS  RESPONDING) 


AGENT  JOB 
SATISFACTION 
RATING 
CATEGORY 

MANUAL 

CCIS 

BEFORE 

SURVEY 

AFTER 

SURVEY 

BEFORE 

SURVEY 

AFTER 

SURVEY 

EXCELLENT 

8 

★ 

13 

19 

GOOD 

65 

★ 

54 

55 

FAIR 

16 

* 

25 

23 

POOR 

11 

* 

8 

3 

TOTAL  AGENTS 
RESPONDING 

37 

★ 

24 

31 

* This  question  was  not  included  on  the  manual  agents' 
after  survey. 
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TABLE  3-12.  FREQUENCY  WITH  WHICH  AGENTS  USE  MAP  BOOKS* 
OR  NO  REFERENCES  IN  PROVIDING  CALLER  RESPONSES 
"BEFORE"  SURVEY  RESULTS 
(%  OF  TOTAL  AGENT  RESPONSES) 


AGENT 

FREQUENCY 

RATING 

CCIS 

MANUAL 

MAP 

BOOKS 

NO 

REFERENCE 

MAP 

BOOKS 

NO 

REFERENCE 

HARDLY  AT  ALL 

0 

4 

2 

8 

SOMETIMES 

20 

56 

9 

23 

FREQUENTLY 

48 

24 

49 

38 

VERY  FREQUENTLY 

32 

16 

40 

31 

TOTAL 

AGENT 

RESPONSES 

25 

25 

47 

47 

* 


Thomas  Guides,  used  for  providing  walking  instructions 
or  locating  caller  addresses. 
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question";  and  only  one  agent  chose  the  response 
"retrieving  the  information  in  reference  material". 

None  of  the  agents  felt  that  using  the  map  book  (Thomas 
Guide)  to, locate  the  caller  or  give  walking  instructions" 
caused  them  any  problem  as  compared  with  the  other 
response  alternatives.  A total  of  14  agents  volunteered 
additional  information  under  the  "other"  response. 
Problems  cited  by  these  agents  covered  a wide  variety 
of  issues,  the  most  frequently  mentioned  of  which  were 
rude  or  abusive  callers  and  management  pressures  for 
higher  productivity.  Other  responses  included  the 
quality  of  the  manual  reference  materials,  the  physical 
layout  of  the  manual  work  area  and  agent  boredom. 

The  question  "What  duties  do  you  like  most  about  your 
job?"  did  not  suggest  any  alternative  answer's  but 
merely  provided  a blank  in  which  an  agent  could  write 
his  or  her  own  response.  A total  of  42  manual  agents 
responded  to  this  question.  By  far,  the  most  common 
answer  was  "helping  people".  Other  responses  included 
"giving  out  accurate  information"  and  "receiving  praise 
from  customers".  Several  agents  also  mentioned  their 
job  security  as  union  members. 

The  question  "What  duties  do  you  dislike  most  about 
your  job?"  produced  a total  of  39  agent  responses. 

Once  again,  no  suggested  responses  were  supplied;  only 
a blank  was  provided  for  agents  to  fill  in  their  own. 

A variety  of  different  dislikes  were  raised  by  the 
agents.  The  most  frequent  response  could  generally  be 
stated  as  "dealing  with  difficult  customers".  Agents 
evidently  experience  considerable  frustration  in  having 
to  be  pleasant  to  callers  who  use  abusive  language  or 
who  fail  to  understand  repeated  instructions.  The  next 
frequent  response  could  be  termed  "management  pressure 
to  achieve  high  call  counts".  Other  management  prac- 
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tices  such  as  logging  agent  time  spent  in  rest  rooms 
and  prohibiting  talking  or  smoking  at  work  stations 
were  also  criticized,  as  was  the  relative  inaccessi- 
bility of  shift  supervisors.  Several  agents  mentioned 
other  job-related  complaints  such  as  monotony,  looking 
up  information  for  foreign  lines  (other  bus  companies 
in  the  Los  Angeles  area  with  which  the  agents  are 
unfamiliar)  and  trying  to  adjust  schedule  information 
to  delays  encountered  in  the  real  world. 

Manual  agents  were  asked  the  question:  "Why  do  you 
prefer  manual  call  processing  over  using  the  CCIS?" 

Since  initial  training  for  CCIS  was  done  on  a voluntary 
basis,  it  was  assumed  that  those  agents  who  did  not 
apply  for  CCIS  training  preferred  manual  data  retrieval. 
For  this  question,  several  alternative  responses  were 
provided,  together  with  a blank  labelled  "Other  (please 
explain)".  The  response  most  frequently  chosen  (by  a 
total  of  20  agents)  was  "I'm  more  effective  using  the 
manual  method".  Five  agents  selected  "negative 
experience  with  CCIS",  while  another  four  agents  selected 
"I'm  more  comfortable  with  manual".  A total  of  21 
agents  chose  to  provide  additional  information  in  the 
"other"  response.  Many  of  these  agents  criticized  the 
CCIS  as  being  too  slow,  not  comprehensive  enough,  or 
subject  to  frequent  changes.  Others  mentioned 
unfamiliarity  with  the  equipment  or  poor  typing  skills 
that  led  to  frequent  errors.  One  agent  criticized  the 
accuracy  of  CCIS  response  screens;  "When  the  CCIS  has 
reached  the  point  of  accuracy  that  you  can,  with  clear 
conscience,  give  information  solely  from  the  screen, 
then  I would  love  to  use  it".  Many  of  the  manual  agents 
stated  that  they  had  merely  missed  out  in  the  sign-up 
process  and  would  like  to  try  the  CCIS. 
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• The  final  open-ended  question  was  "How  can  the  manual 

processing  method  be  improved  to  help  you  in  your  work"?. 
On  this  question,  no  responses  were  suggested;  a blank 
was  provided  for  agent  response.  A total  of  43  out  of 
the  47  agents  filled  in  responses  to  this  question. 
Slightly  over  half  this  total  suggested  that  the  manual 
reference  materials  be  kept  in  good  condition  and  up  to 
date.  Many  agents  complained  that  the  schedule  books 
were  unwieldly  and  in  ragged  or  poor  condition.  Several 
suggested  that  references  be  set  in  larger  print  or 
laid  out  in  a more  readable  fashion.  Others  suggested 
changes  to  the  physical  layout  of  the  information 
center  which  might  reduce  the  number  of  times  agents 
have  to  leave  their  work  stations  to  find  a particular 
reference.  A few  of  the  agents  felt  that  better  manage- 
ment attitudes  towards  the  needs  of  the  agents  would 
improve  the  system.  One  agent  suggested  the  recording 
played  to  callers  while  waiting  for  an  available  agent 
be  changed  so  as  to  better  prepare  callers  to  state- 
their  queries  in  a format  useful  to  the  agent. 

Tables  3-13  and  3-14  present  the  results  of  cross- tabulating 
the  responses  to  certain  of  the  manual  survey  questions.  In 
Table  3-13,  agent  perceptions  of  overall  ability  are  cross- 
tabulated  against  agent  experience.  In  this  table,  it  can  be 
seen  that  agent  perception  of  ability  generally  increases  with 
experience.  Table  3-14  presents  call  count  per  hour  when  busy 
vs.  experience  as  an  agent.  The  distribution  of  responses  on 
this  table  generally  supports  the  notion  that  call  count 
increases  with  experience. 

3. 7. 1.2  CCIS  Agent  "Before"  Survey  Results  - In  their  "before" 
survey,  CCIS  agents  were  asked  a total  of  14  questions,  five  of 
which  permitted  open-ended  responses.  Agent  responses  to  each 
are  summarized  below: 
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TABLE  3-13.  MANUAL  AGENTS  "BEFORE"  SURVEY 
CROSS  TABULATION  OF  CALL  COUNT  VS  EXPERIENCE 


CALL 

WHEN 

COUNT 

BUSY 

AGENT  EXPERIENCE 

TOTAL 

0-6 

MONTHS 

7-12 

MONTHS 

1-2 

YEARS 

2-3 

YEARS 

3 OR 
MORE 
YEARS 

1 5 or 

Le  s s 

1 

1 

0 

0 

0 

2 

16  - 

20 

2 

1 

3 

0 

6 

12 

21  - 

25 

1 

1 

9 

2 

10 

23 

26  - 

30 

0 

0 

6 

0 

2 

8 

Over 

30 

0 

0 

0 

1 

1 

2 

TOTAL 

4 



3 

18 

3 

19 

47 

TABLE  3-14.  MANUAL  AGENTS  "BEFORE"  SURVEY 
CROSS/TABULATION  OF  ABILITY  VS  EXPERIENCE 


ABILITY 

RATING 

AGENT  EXPERIENCE 

0-6  MO. 

7-12  MO. 

1-2  YR. 

2-3  YR. 

OVER  3 YR. 

EXCELLENT 

1 

0 

7 

3 

7 

GOOD 

3 

3 

11 

0 

11 

FAIR 

0 

0 

0 

0 

1 

POOR 

0 

0 

0 

0 

0 

TOTAL 

4 

3 

18 

3 

19 
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As  in  the  manual  survey,  CCIS  agents  categorized 
their  overall  job  experience  and  hourly  call  counts 
under  busy  conditions.  These  responses  are  shown  on 
the  third  columns  of  Tables  3-8  and  3-9.  The  CCIS 
agents  responding  were  as  a group  slightly  more 
experienced;  their  average  job  tenure  was  approxi- 
mately 25  months.  Hourly  call  counts  for  the  CCIS 
agents  were  generally  in  the  upper  teens  to  lower 
twenties,  very  comparable  to  the  manual  agents 
responding . 

CCIS  agents  were  also  asked  in  their  "before"  survey 
to  rate  their  abilities.  Responses  to  this  question 
are  tabulated  in  the  third  column  of  Table  3-10.  It 
is  evident  that  the  CCIS  agents  surveyed  have  a lower 
opinion  of  their  abilities  using  new  technology  than 
do  their  counterparts  using  manual  methods.  This 
result  is  also  reflected  in  CCI'S  agent  responses  to 
the  question  "Do  you  feel  that  you  have  reached  your 
full  potential  in  using  the  CCIS?"  A total  of  75%  of 
the  agents  responded  "No". 

The  third  column  in  Table  3-11  provides  CCIS  agent 
responses  to  the  survey  question  regarding  agent  job 
satisfaction.  The  percentages  of  agents  rating  their 
job  satisfaction  as  "excellent"  or  "good"  are  quite 
comparable  to  those  for  the  manual  agents  in  their 
"before"  survey. 

CCIS  agents  were  also  asked  to  evaluate  the  manner  in 
which  the  CCIS  had  impacted  their  overall  job  satis- 
faction. Responses  to  this  question  are  shown  in 
Table  3-15.  Results  show  that  agents  were  fairly 
evenly  divided  between  the  "positive  influence"  and 
"no  influence"  responses.  Only  one  agent  chose  the 
"negative  influence"  response. 
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TABLE  3-15.  CCIS  AGENT  RESPONSES  TO  SURVEY  QUESTIONS 
REGARDING  CCIS  INFLUENCES  ON  JOB  SATISFACTION 
(%  OF  TOTAL  AGENT  RESPONSES) 


AGENT 

PERCEPTION 
OF  CCIS 
INFLUENCE 

BEFORE 

SURVEY 

AFTER 

SURVEY 

POSITIVE 

52 

84 

NEGATIVE 

4 

0 

NO  INFLUENCE 

48 

16 

TOTAL 

AGENTS 

RESPONDING 

25 

31 
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Table  3-12  displays  CCIS  agent  "before"  survey 
responses  to  questions  concerning  the  frequencies 
with  which  they  consult  their  map  books  (Thomas 
Guides)  and  with  which  they  are  able  to  provide 
caller  responses  without  recourse  to  any  reference 
material.  From  this  table,  it  is  evident  that  CCIS 
agents  responding  to  the  survey  are  just  as  likely  to 
use  the  map  books  as  their  manual  counterparts.  They 
are  less  likely,  however,  to  respond  to  callers  "off 
the  tops  of  their  heads".  Perhaps  this  result  is  due 
to  the  ease  and  speed  with  which  the  CCIS  can  be 
consulted,  as  compared  with  the  manual  reference 
mater ial s. 

The  CCIS  agents,  all  of  whom  had  been  trained  on  and 
used  manual  data  retrieval  methods  before  system 
implementation,  were  asked  to  compare  CCIS  vs.  manual 
call  processing  across  a number  of  different  categories. 
Table  3-16  shows  the  results  of  these  comparisons. 

These  results  indicate  that  the  CCIS  was  rated  very 
favorably  in  a number  of  categories  by  the  CCIS  agents. 
Of  particular  interest  is  the  fact  that  80%  of  these 
agents  felt  that  CCIS  made  their  jobs  easier,  while 
only  three  felt  that  CCIS  was  worse  than  manual  in 
this  respect.  In  the  category  "ability  to  handle 
multiple  requests",  the  CCIS  received  its  most 
unfavorable  rating,  with  32%  of  the  agents  stating 
that  CCIS  was  worse  than  manual  and  an  additional  20% 
more  stating  that  the  two  systems  were  about  the  same 
in  this  respect.  In  the  categories  of  "accuracy"  and 
"caller  satisfaction",  the  "about  the  same"  responses 
outnumbered  "better"  and  "worse"  responses.  Over  all 
the  comparison  categories,  the  CCIS  received  "better" 
ratings  on  about  53%  of  the  responses;  "about  the 
same"  ratings  on  about  31%,  and  "worse"  ratings  on 
only  16%. 
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TABLE  3-16.  CCIS  AGENT  COMPARISONS  OF  MANUAL  VS  CCIS 
INFORMATION  PROCESSING  FOR  VARIOUS  COMPARISON  CATEGORIES 
(%  OF  TOTAL  AGENT  RESPONSES) 


COMPARISON 

CATEGORY 

RATING  CATEGORY 

BET 

TER 

WORSE 

SAME 

BEFORE 

AFTER 

BEFORE 

AFTER 

BEFORE 

AFTER 

RETRIEVAL 

SPEED 

64 

81 

12 

6 

24 

13 

ACCURACY  OF 

INFORMATION 

40 

77 

12 

3 

48 

19 

DETAIL  OF 

INFORMATION 

52 

71 

16 

0 

32 

29 

ABILITY  TO  HANE 

)LE 

MULTIPLE  REQUESTS  48 

55 

32 

26 

20 

19 

MAKING  JOB  EASIER  80 

84 

12 

0 

8 

16 

PROVIDING 

SATISFACTION  TO 

THE  CALLER 

24 

58 

12 

6 

64 

35 

OVERALL  CAPACIT 

Y 60 

77 

16 

0 

24 

23 

PERCENT  OF 

ALL  RESPONSES 

53 

73 

16 

6 

32 

21 

TOTAL  AGENTS 

RESPONDING 

25 

31 

25 

31 

25 

31 
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Several  alternative  answers  were  provided  for  the 
questions  "What  part  of  the  CCIS  causes  you  the  most 
problems?",  as  well  as  a blank  labeled  "other"  for 
agents  to  supply  additional  information  if  they  chose 
to  do  so.  Eight  agents  chose  the  response  "Which  key 
function  to  use";  two  chose  "evaluating  the  infor- 
mation on  the  screen  before  giving  it  to  the  caller", 
and  12  chose  to  supply  their  own  responses  under  the 
"other"  response.  These  12  agents  cited  a variety  of 
system-related  problems,  including  the  following: 

(1)  Not  enough  key  functions  are  supplied,  causing 
agents  to  do  more  typing  than  is  necessary. 

(2)  Translating  customer  requests  into  CCIS  functions. 

(3)  Slow  or  inaccurate  responses,  or  insufficient 
detail . 

(4)  Not  enough  agent  training. 

(5)  Inflexibility  in  handling  multiple  requests, 
causing  the  agent  to  re-enter  functions  numerous 
times. 

CCIS  agent  responses  to  "before"  survey  questions 
regarding  aspects  of  their  jobs  most  liked  and  disliked 
were  very  similar  to  manual  agent  responses.  Regarding 
aspects  liked  most,  the  most  frequent  response  supplied 
by  the  agents  was  "helping  the  public";  this  response 
was  cited  by  32%  of  the  agents.  Other  responses 
included  "intelligent  customers",  "giving  out  useful 
information",  "helping  customers  save  money",  and 
"union  security".  Regarding  aspects  disliked  the 
most,  a total  of  40%  of  the  agents  singled  out 
"dealing  with  difficult  or  abusive  customers".  Other 
dislikes  included  looking  up  information  for  foreign 
lines,  boredom,  dealing  with  management  pressure,  and 
CCIS  breakdowns. 
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• The  CCIS  agents  were  most  responsive  to  the  final 

open-ended  question  "How  can  the  CCIS  be  improved  to 
help  you  in  your  work?".  A total  of  22  out  of  25 
agents  filled  in  suggestions.  The  most  frequent 
response  to  this  questions  was  "expand  the  CCIS 
systemwide".  Other  suggestions  included: 

(1)  Make  the  key  functions  more  flexible. 

(2)  Make  data  on  bus  transfers  easier  to  retrieve. 

One  agent  cited  the  "inability  to  get  next  con- 
nection, first  connection,  and  last  connection 
without  extensive  manipulation  of  time  inputs. " 

(3)  Add  walking  instructions  to  the  system.  The  CCIS 
explicitly  lacks  this  capability  at  present;  for 
this  reason  the  agents  must  consult  their  map 
books  when  giving  walking  instructions. 

(4)  Increase  response  accuracy. 

(5)  Add  smaller  streets  to  the  dictionary  of  recog- 
nized landmarks. 

(6)  Speed  up  system  response  time. 

(7)  Add  a backup  computer  to  the  system  to  reduce 
down  time. 

(8)  Increase  initial  agent  training. 

Tables  3-17  and  3-18  illustrate  the  results  of  two  cross- 
tabulations run  on  CCIS  "before"  survey  results.  Table  3-17 
contains  data  for  CCIS  agents  similar  to  data  in  Table  3-13  for 
manual  agents,  and  cross- tabulates  agent  call  counts  vs.  job 
experience.  The  pattern  of  responses  in  this  table  supports 
the  general  notion  that  agent  call  count  increases  with  experience. 
Table  3-18  plots  the  frequency  with  which  agents  use  no  reference 
materials  in  response  to  caller  queries  vs.  agent  experience. 

While  the  distribution  of  responses  in  this  table  is  rather 
scattered,  it  generally  confirms  the  idea  that  increased 
experience  is  correlated  with  increased  agent  ability  to 
respond  "off  the  tops  of  their  heads". 
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TABLE  3-17.  CROSS-TABULATION  OF  AGENT  CALL  COUNT 
VS  JOB  EXPERIENCE  RESPONSES 
CCIS  AGENT  "BEFORE"  SURVEY 


CALL  COUNT 
PER  HOUR 
WHEN  BUSY 

EXPERIENCE  AS  AN  AGENT 

TOTAL 

0-6 

MO. 

7-12 

MO. 

1-2 

YR. 

2-3 

YR. 

OVER 
3 YR. 

15  OR  LESS 

0 

0 

0 

0 

0 

0 

16-20 

1 

0 

2 

3 

3 

9 

21-25 

0 

0 

6 

1 

6 

13 

26-30 

0 

0 

C 

0 

2 

2 

OVER  30 

0 

0 

0 

0 

1 

1 

TOTAL 

1 

0 

8 

4 

12 

25 
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TABLE  3-18.  CROSS-TABULATION  OF  "NO  REFERENCE" 
RESPONSE  FREQUENCY  VS  AGENT  JOB  EXPERIENCE 
CCTS  AGENTS  "BEFORE"  SURVEY 


FREQUENCY 

EXPERIENCE  AS  AN  AGENT 

Ur  [nU 

REFERENCE" 

RESPONSES 

0-6 

MO. 

7-12 

MO. 

1-2 

YR. 

2-3 

YR. 

OVER 
3 YR. 

TOTAL 

HARDLY 
AT  ALL 

0 

0 

0 

0 

1 

1 

SOMETIMES 

1 

0 

5 

3 

5 

14 

FREQUENTLY 

0 

0 

2 

0 

4 

6 

VERY 

FREQUENTLY 

0 

0 

1 

1 

2 

4 

TOTAL 

1 

0 

8 

4 

12 

25 
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3.7.2  "After"  Agent  Survey  Results 

The  "after"  surveys  administered  to  both  CCIS  and  manual 
agents  were  very  similar  in  both  form  and  content  to  the 
"before"  surveys.  In  some  instances,  questions  identical  to 
those  in  the  "before"  surveys  were  asked  of  each  group.  Once 
again,  agents  were  highly  cooperative  in  completing  the  survey; 
a total  of  30  manual  and  31  CCIS  surveys  were  returned  for 
processing.  Responses  were  coded  and  analyzed  using  SPSS 
methods  similar  to  those  used  in  the  "before"  survey. 

3.7. 2.1  Manual  Agent  "After"  Survey  Results  - In  their  "after" 
survey,  manual  agents  were  asked  a total  of  11  questions,  two 
of  which  allowed  open-ended  responses.  Agent  responses  are 
presented  below. 

• As  in  the  "before"  surveys,  manual  agents  were  asked  to 
report  their  job  tenure  and  call  count  under  busy 
conditions.  Their  responses  are  shown  in  the  second 
columns  of  Tables  3-8  and  3-9.  Once  again,  manual 
agents  responding  to  the  survey  were  generally  well- 
experienced,  with  call  counts  in  the  upper  teens  to 
lower  twenties  per  hour  on  average.  These  results  are 
very  similar  to  the  manual  agent  "before"  survey 
responses. 

• Manual  agents  were  again  requested  to  rate  their  over- 
all abilities.  Their  responses  to  this  survey  question 
are  shown  in  the  second  column  of  Table  3-10.  Manual 
agent  self-assessments  were  once  again  quite  high,  with 
over  half  the  respondents  classifying  themselves  in  the 
"excellent"  category,  and  all  but  one  of  the  remainder 
rating  themselves  "good". 

All  the  manual  agents  had  presumably  had  the  opportunity  at 
some  point  during  the  90-day  test  period  to  at  least  observe  the 
CCIS  in  operation,  if  not  to  try  it  out  for  themselves.  A total 
of  20  of  the  30  manual  agents  responding  to  the  "after"  survey 
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indicated  they  had  received  instruction  in  CCIS  use  as  part  of 
the  revised  agent  training  program  instituted  by  SCRTD  manage- 
ment during  this  period.  As  a result,  it  was  assumed  that  the 
manual  agents  had  formed  some  opinions  regarding  CCIS,  and  a 
series  of  questions  on  these  agents'  "after"  survey  attempted  to 
document  these  opinions. 

• To  the  question  "Which  do  you  feel  is  the  best  method 
of  information  processing?"  only  13%  of  the  manual 
agents  replied  "manual  only";  seven  replied  "CCIS 
only";  53%  replied  "mixture  of  manual  and  CCIS";  and 
10%  replied  "don't  know". 

• When  asked  "Would  you  like  to  learn  to  use  CCIS?",  73% 
of  the  agents  replied  "yes",  7%  replied  "no"  and  17% 
replied  "don't  know".  A follow-up  question  asked 
agents  to  explain  this  response  in  an  open-ended 
fashion.  Explanations  for  agent  desire  to  learn  CCIS 
included  "improves  call  count"  (17%);  "increases  job 
skills"  (10%);  "makes  the  job  better"  (17%).  Explana- 
tions for  agent  aversion  to  learning  CCIS  included 
"poor  typing  skills"  (7%)  and  "problems  with  CCIS 
accuracy"  ( 13% ) . 

• When  asked  "Do  you  feel  that  learning  to  use  CCIS 
would  increase  your  job  satisfaction?",  60  of  the 
agents  responded  "yes",  while  20%  responded  "no"  and 
20%  responded  "don't  know". 

• Manual  agents  were  asked  to  predict  how  learning  to 
use  CCIS  would  affect  their  hourly  call  counts. 

Survey  responses  to  this  question  are  shown  in  Table 
3-19.  A total  of  50%  of  the  manual  agents  felt  their 
call  counts  would  increase,  14%  felt  that  they  would 
remain  the  same,  and  36%  felt  that  they  would  probably 
decrease.  This  response  indicates  at  least  some 
apprehension  on  the  part  of  the  manual  agents  con- 
cerning their  abilities  to  interact  effectively  with 
the  computer. 
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TABLE  3-19.  "AFTER"  SURVEY  RESPONSES  TO  QUESTIONS 
REGARDING  INFLUENCE  OF  CCIS  ON  AGENTS'  CALL  COUNT 
(%  OF  TOTAL  AGENTS  RESPONDING) 


AGENT 
PERCEPTION 
OF  CCIS 
INFLUENCE 

MANUAL  AGENTS* 

CCIS  AGENTS** 

INCREASE 

50 

45 

SAME 

14 

41 

DECREASE 

36 

14 

TOTAL 

AGENTS 

RESPONDING 

28 

29 

* "How  do  you  feel  that  learning  to  use  the  CCIS  would 
affect  your  hourly  call  count?" 

**.  "How  has  using  the  CCIS  affected  your  hourly  call 
count?" 
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• The  final  two  questions  on  the  manual  survey  dealt 
with  manual  agent  opinions  concerning  the  proposed 
expansion  of  the  CCIS  to  cover  the  entire  SCRTD 
service  area.  A total  of  75%  of  the  manual  agents 
expressed  favorable  responses,  while  only  14%  were 
opposed.  Another  11%  replied  "don't  care".  When 
asked  to  explain  this  response  in  an  open-ended 
question,  a total  of  22  agents  provided  responses.  Of 
these,  eight  emphasized  that  CCIS  retrieval  speed  was 
much  faster  than  manual,  five  cited  CCIS  response 
accuracy,  and  three  stated  that  the  proposed  expansion 
would  improve  their  jobs.  Of  those  agents  explaining 
their  opposition  to  the  proposal,  four  mentioned 
problems  with  CCIS  accuracy  and  two  others  noted  that 
they  had  poor  typing  skills. 

3. 7. 2. 2 CCIS  Agent  "After"  Survey  Results  - In  their  "after" 
survey,  CCIS  agents  were  asked  a total  of  13  questions,  of  which 
3 permitted  open-ended  responses.  Agent  responses  are  presented 
below. 

• CCIS  agents  were  also  requested  to  again  report  their 
job  tenure  and  call  count  under  busy  conditions. 

Their  responses  are  shown  in  the  fourth  columns  of 
Tables  3-8  and  3-9.  Responses  were  very  similar  to 
previous  manual  and  CCIS  surveys.  On  the  whole,  the 
CCIS  respondents  were  more  experienced  than  the 
manual:  union  seniority  giving  day-shift  preference  to 
the  more  experienced  agents  is  one  explanation  for 
this  result.  Call  counts  reported  by  the  CCIS  agents 
were  once  again  at  or  near  20  per  hour  on  average,  the 
SCRTD  standard. 

• CCIS  agents  were  again  asked  to  rate  their  abilities 
in  using  the  system.  Their  self-assessments  are  shown 
in  the  fourth  column  of  Table  3-10.  It  is  evident 
from  these  data  that  the  CCIS  agents  became  more 
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comfortable  with  automated  information  retrieval 
during  the  90-day  test  period,  as  the  percentage  of 
agents  rating  their  abilities  as  "excellent"  or  "good" 
increased  by  30%  over  the  "before"  survey  results. 
Because  the  "before"  survey  had  shown  such  marked 
differences  in  ability  ratings  between  manual  and  CCIS 
agents,  the  CCIS  "after"  survey  also  included  a 
question  asking  the  CCIS  agents  to  rate  their  abilities 
using  manual  retrieval.  These  ratings  are  shown  in 
the  fifth  column  of  Table  3-10.  They  are  interesting 
in  that  they  indicate  the  CCIS  agents  still  considered 
themselves  considerably  "better"  at  using  manual 
retrieval  than  at  using  the  CCIS.  It  should  be  noted, 
however,  that  half  of  the  CCIS  agents  responding  had 
been  using  the  system  for  less  than  six  months  and 
were  probably  not  yet  fully  proficient  using  CCIS. 

The  CCIS  "after"  survey  also  contained  two  questions 
pertaining  to  agent  job  satisfaction.  The  first  of 
these  questions  asked  for  a general  opinion  of  overall 
agent  job  satisfaction.  Responses  to  this  question 
are  shown  in  the  fourth  column  of  Table  3-11.  These 
responses  are  only  marginally  different  from  the 
"before"  survey  responses,  and  indicate  a generally 
high  level  of  satisfaction  among  CCIS  agents.  Only 
one  agent  responded  "poor"  to  this  question.  The 
second  question  asked  for  agent  opinion  concerning  the 
influence  of  the  CCIS  on  the  agents'  job  satisfaction. 
Responses  to  this  question  are  shown  in  the  second 
column  of  Table  3-15.  These  responses  are  consi- 
derably more  supportive  of  the  CCIS  than  those  of  the 
CCIS  "before"  survey,  and  reflect  a shift  of  opinion 
for  over  30%  of  the  agents  from  the  "negative"  or  "no 
influence"  to  the  "positive"  response. 
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CCIS  agents  were  also  asked  in  the  "after  survey  to 
characterize  the  influence  of  system  implementation  on 
their  productivity  as  measured  by  hourly  call  counts. 
Responses  to  this  question  are  shown  in  the  second 
column 'of  Table  3-16.  These  results  indicate  that 
about  the  same  percentage  of  CCIS  agents  as  manual 
agents  felt  that  CCIS  had  increased  or  would  increase 
their  call  counts.  About  25%  more  of  the  CCIS  group, 
however,  felt  that  the  system  had  no  influence  on 
their  call  counts  as  opposed  to  the  decrease  foreseen 
by  the  manual  agents. 

CCIS  agents  were  again  asked  to  compare  CCIS  versus 
manual  call  processing  over  a variety  of  different 
rating  categories.  This  survey  question  was  identical 
to  one  asked  in  the  "before"  survey.  Comparison  of 
the  "before"  and  "after"  responses  to  this  question 
may  thus  provide  insights  into  changes  in  agent 
attitudes  towards  the  system  which  may  have  developed 
during  the  course  of  the  90-day  test  period.  The 
second,  fourth  and  sixth  columns  in  Table  3-16  show 
the  "after"  survey  responses  to  this  question.  Com- 
parison of  These  responses  to  the  "before"  responses 
indicates  that  CCIS  agents  once  again  rated  the  system 
very  favorably;  in  fact,  the  ratings  they  assigned  are 
generally  higher  than  those  given  in  the  "before" 
survey.  All  of  the  agents  responding  to  this  question 
rated  the  CCIS  as  "about  the  same"  or  "better"  in  the 
category  "making  your  job  easier".  Once  again,  the 
CCIS  received  its  lowest  overall  rating  in  the  category 
"ability  to  handle  multiple  requests";  with  36%  of  the 
agents  stating  that  CCIS  was  worse  than  manual  and  an 
additional  19%  stating  that  the  two  retrieval  modes 
were  about  the  same  in  this  regard.  In  the  categories 
of  "accuracy"  and  "caller  satisfaction",  agents  this 
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time  rated  CCIS  better  than  manual  by  sizable  margins. 
These  responses  represent  a substantial  shift  from  the 
"before"  survey  in  which  the  "about  the  same"  responses 
outnumbered  the  "better"  and  "worse"  responses  in 
these  two  rating  categories.  Over  all  comparison 
categories,  the  CCIS  was  rated  "better"  than  manual  by 
73%  of  agent  responses;  "about  the  same"  by  about  21%, 
and  "worse"  by  only  6%. 

A similarly  high  rating  of  the  system  was  indicated  by 
CCIS  agents  in  response  to  the  question  "What  aspect 
of  the  CCIS  causes  you  the  greatest  problem?".  In 
response  to  this  question,  agents  were  given  a choice 
of  four  "problems"  which  had  been  frequently  mentioned 
in  the  "before"  surveys,  plus  a blank  labeled  "other" 
for  a more  detailed  open-ended  response.  The  most 
frequent  response  to  this  question,  given  by  a total 
of  11  agents,  was  "no  problems"  entered  in  the  "other" 
category.  Problems  cited  by  other  respondents 
included  "typing  in  caller  information"  (9  agents)  and 
completeness/accuracy  of  CCIS  responses"  (8  agents). 
Only  two  agents  responded  "choosing  which  function  to 
use",  a considerable  decrease  in  this  category  from 
the  results  of  a similar  question  in  the  "before" 
survey.  Perhaps  this  difference  indicates  an  increase 
in  agent  familiarity  with  the  various  key  functions 
and  their  capabilities  and  limitations. 

CCIS  agent  responses  to  the  open-ended  question  "How 
can  CCIS  be  improved  to  help  in  your  work?"  can  be 
grouped  into  two  categories.  The  first  includes  those 
of  ten  agents  who  suggested  improvements  to  the  key 
functions  and  other  machine  capabilities  such  as: 
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(1)  'Scrolling'  schedule  screens  which  enable  an 
agent  to  see  considerably  more  schedule  infor- 
mation on  the  response  screen  for  a given  bus 
route  than  is  ordinarily  provided  by  a schedule 
function  reference. 

(2)  Addition  of  at  least  rudimentary  walking  instruction 
capabil i ties. 

(3)  Changing  the  schedule  function  so  that  agents  can 
compare  schedules  of  two  alternative  bus  routes 
displayed  at  the  same  time  on  the  screen. 

(4)  Temporary  storage  capabilities  for  itinerary 
information  so  it  can  be  retrieved  without  delay 

if  other  functions  are  also  called  up.  At  present, 
use  of  other  functions  "erases"  this  information 
from  the  response  screen. 

(5)  Adding  information  section  paperwork  functions  to 
the  CCIS  software;  i.e.  time  sheets,  progress 
reports,  agent  bidding  for  shifts,  vacation  times, 
etc . 

The  second  response  category,  which  included  13  agents, 
consisted  of  suggestions  to  expand  the  geographic 
scope  of  the  CCIS  to  include  the  entire  SCRTD  service 
area. 

As  in  the  "after"  manual  survey,  the  final  two  questions 
on  the  "after"  CCIS  survey  dealt  with  agent  opinions 
concerning  the  proposed  expansion  of  the  geographic 
scope  of  the  CCIS.  CCIS  agents  overwhelmingly  favored 
this  proposal:  94%  of  the  agents  replied  "favor  the 
idea"  while  only  3%  replied  "oppose  the  idea"  and  3% 
replied  "no  opinion". 

When  asked  to  explain  this  response  in  an  open-ended 
follow-up  question,  24  agents  responded.  Of  these, 
nine  cited  CCIS  retrieval  speed,  five  mentioned  CCIS 
accuracy,  five  stated  that  it  made  their  jobs  easier, 
four  noted  increased  caller  satisfaction,  and  one 
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mentioned  that  it  helped  develop  new  job  skills. 

Tables  3-20  through  3-22  present  results  of  cross- tabula- 
tions of  interest  run  on  various  responses  to  the  CCIS  "after" 
survey.  Table  3-20  presents  a cross- tabulation  of  CCIS  influence 
on  call  count  responses  vs.  agent  experience  responses.  This 
table  shows  that  CCIS  had  a greater  positive  influence  on  the 
call  counts  of  agents  with  less  than  3 years  experience  than 
those  agents  having  over  3 years  of  experience.  Table  3-21 
presents  a similar  cross- tabulation,  this  time  showing  CCIS 
experience  versus  CCIS  influence  on  call  count.  From  this 
table,  it  can  be  seen  that  a greater  percentage  of  those  agents 
recently  introduced  to  the  CCIS  (less  than  3 months  experience) 
claimed  reductions  in  their  overall  call  counts  than  did  agents 
having  greater  CCIS  experience. 

Table  3-22  shows  "experience  using  CCIS"  responses  tabu- 
lated against  "problems  using  CCIS  responses".  This  table 
shows  that  "choosing  which  function  key  to  use"  was  cited  by 
only  two  less-experienced  agents,  while  "typing  in  caller 
information"  and  "CCIS  accuracy"  are  problems  noted  at  all 
experience  levels.  "No  problems"  was  another  response  cited  by 
agents  of  all  CCIS  experience  levels. 

3.8  CCIS  CONTROLLED  EXPERIMENT  RESPONSE  QUALITY  RESULTS 

All  324  agent  responses  logged  during  the  course  of  the 
CCIS  controlled  experiment  were  graded  by  SCRTD  Planning 
Department  personnel  and  by  the  Supervisor  of  Training  in  the 
Information  Section.  In  the  grading  process,  major  emphasis 
was  placed  on  the  accuracy  and  completeness  of  each  response. 

To  judge  accuracy  and  completeness,  each  agent  response  was 
compared  with  a "best"  itinerary  or  other  response  as  deter- 
mined independently  by  the  graders.  The  determination  of  this 
"best"  response  was  made  using  SCRTD  manual  reference  materials 
available  on  the  date  of  the  experiment.  Since  the  CCIS  data 
base  relied  on  these  same  manual  materials  for  updating,  it  was 


-111- 


TABLE  3-20.  CROSS-TABULATION  OF  AGENT  PERCEPTIONS 
REGARDING  CCIS  INFLUENCE  ON  CALL  COUNTS  VS  JOB  EXPERIENCE 

CCIS  AGENT  "AFTER"  SURVEY 


CCIS  INFLUENCE 
ON  CALL  COUNT 

EXPERIENCE  AS  AN 

AGENT 

1-2 

YRS 

2-3 

YRS 

OVER 
3 YRS 

REDUCTION 

1 

0 

3 

NO  INFLUENCE 

3 

2 

7 

INCREASE 

3 

6 

4 
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TABLE  3-21.  CROSS-TABULATION  OF  AGENT  RESPONSES  REGARD] NG 
EXPERIENCE  USING  CCIS  VS  AGENT  PERCEPTIONS  OF 
CCIS  INFLUENCE  ON  HOURLY  CALL  COUNT 
CCIS  AGENT  "AFTER"  SURVEY 


AGENT 

EXPERIENCE 
USING  CCIS 

CCIS  INFLUENCE  ON  CALL 

COUNT 

REDUCTION 

NO 

INFLUENCE 

INCREASE 

OVER  6 MONTHS 

1 

7 

8 

3-6  MONTHS 

1 

4 

3 

3 MONTHS  OR  LESS 

2 

1 

2 
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TABLE  3-22.  CROSS-TABULATION  OF  AGENT  RESPONSES  REGARDING 
EXPERIENCE  USING  CCIS  VS  AGENT  RESPONSES  CONCERNING 

PROBLEMS  USING  CCIS 
CCIS  AGENT  "AFTER "SURVEY 


AGENT 

EXPERIENCE 
USING  CCIS 

PROBLEMS  USING  CCIS 

FUNCTION 

KEYS 

TYPING 

ERRORS 

ACCURACY 
OF  CCIS 

NO 

PROBLEM 

OVER  6 MONTHS 

0 

5 

4 

7 

3-6  MONTHS 

1 

3 

2 

1 

3 MONTHS  OR  LESS 

1 

1 

2 

3 
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assumed  that  the  manual  and  CCIS  data  bases  contained  comparable 
information  on  the  date  of  the  experiment.  The  primary  criterion 
for  judging  the  accuracy  of  each  response  focused  on  the  question 

"Using  the  response  information  provided  by  the  agent,  can 

the  caller  actually  make  a trip  from  Point  A to  Point  B in 

view  of  query  input?" 

The  answer  to  this  question  provided  a simple  and  decisive 
standard  for  determining  overall  response  quality.  If  the 
caller  could  indeed  make  the  trip,  the  response  was  rated 
satisfactory  (S) . If  the  caller  could  not  make  the  trip,  the 
response  was  rated  unsatisfactory  (U) . 

Assessing  the  completeness  of  the  information  provided  in 
each  response  focused  on  those  itinerary  instructions  volun- 
teered to  the  agent.  Items  emphasized  included  the  following: 

• bus  route  selection 

• schedule  information 

• transfer  information 

• fare  information 

• walking  instructions  (if  requested) 

All  those  responses  assigned  satisfactory  (S)  ratings  in 
the  accuracy  grading  were  evaluated  for  completeness  using 
these  criteria.  They  were  assigned  a numerical  rating  based  on 
a progressive  scale  of  one  (minimal  information  provided)  to 
ten  (all  necessary  information  provided).  A response  with  an 
"S-10"  rating  was  therefore  comparable  to  the  "best"  response 
as  determined  by  the  SCRTD  graders. 

A review  of  agent  response  grades,  which  are  presented  in 
Appendix  C,  reveals  that  there  was  some  variability  to  grades 
assigned,  but  the  preponderance  of  these  grades  fell  within  the 
S-6  to  S-10  range.  Responses  in  this  range,  which  accounted 
for  79%  of  the  total,  might  be  termed  "useful".  Responses 
assigned  grades  in  the  S-1  to  S-5  range,  which  might  be 
characterized  as  "marginal",  accounted  for  an  additional  12%  of 
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the  total.  Unsatisfactory  grades  were  assigned  in  28  cases,  or 
9%  of  the  total. 

From  the  frequencies  of  unsatisfactory  and  marginal  grades 
assigned,  as  shown  in  Table  3-23,  it  can  be  seen  that  Question 
Sets  I and  II  were  found  to  be  the  more  difficult  by  the  test 
agents.  These  question  sets  accounted  for  67%  of  all  marginal 
responses  and  75%  of  all  unsatisfactory  responses. 

Comparison  of  agent  grades  received  broken  down  by  skill/ 
experience  level,  shown  in  Table  3-24,  demonstrates  important 
differences  between  the  three  groups.  As  might  be  expected, 
the  agents  of  the  advanced  group  scored  considerably  higher 
than  did  the  other  two  groups:  86%  of  all  their  responses  were 
in  the  useful  range.  The  intermediate  agents  were  next  best  at 
providing  accurate  responses  with  79%  of  their  responses  in 
this  range.  Comparison  of  unsatisfactory  grades  by  each  group 
reveals  a reverse  ranking. 

Comparisons  of  agent  response  grades  by  mode  of  data  retrieval 
also  reveal  interesting  differences  among  the  three  modes.  Table 
3-25  shows  the  relative  frequencies  of  useful,  marginal  and  ' 
unsatisfactory  grades  by  mode  of  data  retrieval.  The  agents  in 
the  mixed  mode  group,  who  were  allowed  to  use  either  CCIS,  manual 
or  a combination  of  both,  were  clearly  rated  the  best  at  providing 
the  most  accurate  and  complete  responses.  The  mixed  mode  group 
received  the  most  useful  (S-6  to  S-10)  ratings  of  the  three 
groups,  with  83%  of  its  total  receiving  grades  within  this  cate- 
gory. The  mixed  mode  group  also  received  the  lowest  overall 
unsatisfactory  rating,  with  only  21  responses,  or  19.4%  of  its 
total,  in  this  category. 

Examination  of  Table  3-25  shows  that  the  manual  and  CCIS 
modes  are  very  evenly  matched  in  terms  of  response  quality,  with 
the  CCIS  slightly  behind  in  terms  of  useful  and  unsatisfactory 
grades,  but  ahead  in  terms  of  marginal  grades. 
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TABLE  3-23.  FREQUENCY  OF  RESPONSE  QUALITY  GRADES 
RECEIVED  BY  ALL  TEST  AGENTS 


RESPONSE 

GRADE 

QUESTION 

SET 

TOTAL 

% OF  TOTAL 

I 

II 

III 

IV 

S-10 

6 

6 

18 

8 

38 

12 

S-9 

12 

19 

26 

25 

82 

25 

S-8 

22 

8 

15 

20 

65 

20 

S-7 

10 

13 

11 

13 

47 

15 

S-6 

9 

9 

2 

4 

24 

7 

S-5 

7 

7 

3 

3 

20 

6 

S-4 

5 

3 

2 

2 

12 

4 

S-3 

2 

0 

0 

3 

5 

2 

S-2 

1 

0 

0 

0 

1 

0 

S-1 

1 

0 

0 

0 

1 

0 

U 

5 

16 

4 

3 

28 

9 
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TABLE  3-24.  RELATIVE  FREQUENCY  OF  RESPONSE  QUALITY  GRADES 
RECEIVED  GROUPED  BY  AGENT  SKILL/EXPERIENCE  LEVEL 
(%  OF  TOTAL  FOR  EACH  GROUP) 


EXPERIENCE/SKILL  LEVEL 

GRADE  ASSIGNED 

NOVICE 

INTERMEDIATE 

ADVANCED 

USEFUL 

(S-6  to  S-10) 

73.1 

78.7 

86.1 

MARGINAL 
(S-1  to  S-5) 

13.9 

13.9 

8.3 

UNSATISFACTORY 

(U) 

13.0 

7.4 

5.6 
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TABLE  3-25.  RELATIVE  FREQUENCY  OF  RESPONSE  QUALITY  GRADES 
RECEIVED  GROUPED  BY  MODE  OF  DATA  RETRIEVAL 


MODE  OF  DATA  RETRIEVAL 

GRADE  ASSIGNED 

MANUAL 

CCIS 

MIXED 

Useful 

(S-6  to  S-10) 

77.8 

76.9 

83.3 

Marginal 
(S-1  to  S-5) 

13.9 

10.2 

12.0 

Unsatisfactory 

8.  3 

12.9 

4.7 
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Table  3-26  provides  a comparison  of  individual  response 
quality  ratings  for  each  agent  participating  in  the  experiment. 

As  can  be  seen  in  this  table,  the  novice  CCIS  agent  clearly 
scored  the  poorest  of  all  the  agents,  with  only  65%  of  his 
responses  falling  into  the  useful  category.  The  manual  and 
mixed  novice  and  the  manual  intermediate  might  be  grouped  next, 
with  between  70%  and  78%  useful  responses.  The  advanced  manual, 
intermediate  CCIS  and  mixed  and  advanced  CCIS  agents  rank  next, 
with  83%  to  86%  of  their  responses  in  this  category.  The 
advanced  mixed  mode  agent  clearly  scored  the  highest,  with  89% 
useful  responses  and  only  one  unsatisfactory  response. 

The  novice  computer  mode  agent  did  so  poorly  on  the  test 
that  it  would  seem  reasonable  to  hypothesize  that  he  had  not  yet 
fully  mastered  the  use  of  the  terminal  keyboard,  the  format 
screens,  or  the  full  range  of  computer  transactions  available 
through  CCIS.  His  response  quality  scores  are  so  low  that  they 
tend  to  diminish  the  overall  showing  of  the  CCIS  agents  as  a 
group,  particularly  in  light  of  the  fact  that  the  novice  agents 
for  the  other  two  modes  posted  fairly  respectable  response 
quality  scores.  Perhaps  a fairer  comparison  between  the  three 
modes  could  be  achieved  by  eliminating  novice  agent  scores  from 
consideration,  and  using  only  intermediate  and  advanced  agent 
scores.  The  results  of  such  a comparison  are  shown  in  Table 
3-27,  which  presents  relative  frequencies  of  grades  received 
grouped  by  mode  with  novices  excluded. 

As  can  be  seen  by  comparing  this  table  with  Table  3-25, 
elimination  of  novice  agent  scores  makes  no  difference  to  the 
percentage  of  useful  responses  in  the  manual  agent  group.  It 
does,  however,  raise  the  useful  percentage  of  both  the  CCIS  and 
mixed  mode  groups  to  a level  where  there  is  little  difference 
between  them.  The  mixed  mode  group,  however,  still  ranks  highest 
overall  in  terms  of  fewest  unsatisfactory  responses.  This  result 
is  somewhat  expected  in  light  of  the  considerably  longer  average 
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TABLE  3-26.  FREQUENCY  OF  RESPONSE  QUALITY 
GRADES  RECEIVED  BY  EACH  TEST  AGENT 


MODE 

RESPONSE 

QUALITY 

MANUAL 

CCIS 

MIXED 

GRADE 

ASSIGNED 

NOV 

INT 

ADV 

NOV 

INT 

ADV 

NOV 

INT 

ADV 

USEFUL 

(S-6  to  S-10) 

28 

25 

31 

23 

30 

30 

28 

30 

32 

MARGINAL 
(S-1  to  S-5) 

5 

7 

3 

5 

3 

3 

5 

5 

3 

UNSATIS- 

FACTORY 

3 

4 

2 

8 

3 

3 

3 

1 

1 
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TABLE  3-27.  RELATIVE  FREQUENCY  OF  RESPONSE  QUALITY 
GRADES  RECEIVED  GROUPED  BY  MODE  OF  DATA  RETRIEVAL 
NOVICE  AGENTS  EXCLUDED 
(%  OF  TOTAL  FOR  EACH  GROUP) 


RESPONSE  QUALITY 
GRADE  ASSIGNED 

MODE  OF  DATA  RETRIEVAL 

MANUAL 

CCIS 

MIXED 

USEFUL 

77.8 

83.3 

86.1 

MARGINAL 

13.9 

8.3 

11.1 

UNSATISFACTORY 

8.3 

8.3 

2.8 
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call  times  of  the  mixed  mode  group  as  compared  to  the  other  two 
modes  of  data  retrieval;  i.e.,  they  took  extra  time  to  double- 
check their  responses  between  the  two  modes. 

3.9  SUMMARIES  OF  INTERVIEWS  WITH  KEY  SCRTD  PERSONNEL 

As  part  of  the  overall  CCIS  evaluation,  a number  of  inter- 
views were  conducted  with  SCRTD  personnel  who  were  instrumental 
in  system  implementation  and  who  managed  system  operations  on  a 
day-to-day  basis  throughout  the  90-day  test  period.  These 
personnel,  grouped  by  SCRTD  Department,  included: 

• Telephone  Information  Section; 

Chief 

Senior  Supervisor 

Senior  Instructor 

CCIS  Program  Manager 

• Customer  Relations 

Director 

• Scheduling  Department 

Advance  Planner 

• Stops  and  Zones 

Supervisor 

• Planning  Department 

Director  and  Chief  Planner 

While  the  content  of  each  "before"  and  "after"  interview  varied, 
an  attempt  was  made  to  focus  on  the  interviewees'  perceptions  of 
CCIS  impacts  on  their  jobs  and  on  the  jobs  of  their  subordinates 
These  impact  areas  were  often  discussed  by  the  interviewees  more 
in  terms  of  projected  impacts  that  in  actual  verified  impacts. 
Their  viewpoints  reflected  the  pilot  nature  of  the  project  and 
the  fact  that  the  SCRTD  was  just  "getting  up  to  speed"  in  terms 
of  steady-state  pilot  CCIS  operation  at  the  time  the  interviews 
were  conducted.  In  a sense  the  interviewees  said  "based  on  what 
we  see  now,  these  are  the  real  impact  areas".  Interviewee 
perceptions  varied  according  to  familiarity  with  CCIS  operations 
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Some  of  those  interviewed  gave  detailed  reactions  to  the  CCIS, 
some  speculated  on  possible  uses  and/or  impacts,  and  others  knew 
very  little  about  the  system. 

3.9.1  Telephone  Information  Section 

3. 9. 1.1  The  Chief  of  the  Telephone  Information  Section  felt 
that  the  CCIS  may  provide  more  administrative  and  managerial 
flexibility  by  decreasing  time  dedicated  to  personnel  scheduling 
shift  assignments,  policies,  procedures  and  disciplinary  actions 
Regarding  the  impacts  felt  by  the  agents,  the  Chief  stated  that 
the  CCIS  should  be  able  to  relieve  some  of  the  mental  stress 
involved  in  manual  information  retrieval;  this  may  help  reduce 
agent  fatigue. 

The  Chief  also  stated  that  the  implementation  of  the  CCIS 
would  have  a major,  beneficial  effect  on  the  job  of  Information 
Section  supervisors.  At  present,  many  supervisors  are  opera- 
tionally "rusty"  due  to  their  unavailability  to  the  agents  in 
terms  of  assisting  with  call  processing  problems  or  work 
procedures.  Almost  all  the  supervisor's  time  is  presently  • 
consumed  in  updating  the  printed  manual  reference  material  in 
the  Summary  Books  at  each  agent's  work  station.  The  CCIS,  by 
incorporating  automated  data  base  updating,  may  free  super- 
visors to  return  to  their  intended  role  as  information  systems 
managers  and  monitors,  and  as  guidance  personnel  available  to 
the  agents.  Less  clerical  responsibility  for  the  supervisors 
would  also  provide  more  time  for  enrolling  them  in  outside 
training  courses  (e.g.,  management  seminars)  for  further 
professional  development.  The  CCIS  would  also  require  a 
different,  more  trouble-shooting  type  of  in-depth  training 
course  for  supervisor  applicants. 

CCIS  implementation  would  also,  the  Chief  mentioned,  have 
a significant  impact  on  Information  Section  recruitment  and 
hiring  practices.  The  Chief  felt  that  the  CCIS  may  make  it 
easier  to  recruit  both  older  personnel  and  part-time  workers 
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for  the  position  of  transit  information  agent.  Older  personnel 
(e.g.,  middle-aged  and  up)  have  been  difficult  to  recruit  in 
the  past  due  to  the  intensive  memorization  requirements  in 
the  training  course;  experience  with  these  personnel,  however, 
has  shown  them  to  be  the  most  reliable  and  competent  performers. 
Hiring  part-time  workers  has  been  difficult  due  to  the  eight 
to  nine  week  training  course  and  full-time  scheduling;  many 
young  college  students  (many  with  related  skills)  could  perform 
as  agents  on  a part-time  basis.  Use  of  part-time  workers 
would  also  provide  more  flexibility  in  scheduling  vacations 
and  extended  holiday  weekends  for  full-time  senior  personnel. 

Finally,  the  Chief  mentioned  that  the  CCIS,  if  implemented 
on  a metropolitan  areawide  basis,  may  enable  the  SCRTD  to 
provide  information  on  other  bus  services  such  as  those  in 
Long  Beach,  Santa  Monica  and  Gardena.  The  Telephone  Informa- 
tion Section  presently  answers  numerous  calls  dealing  with 
these  "foreign"  lines  with  little  assurance  that  the  infor- 
mation regarding  them  is  updated  or  accurate.  The  CCIS  might 
provide  the  means  for  centralizing  information  on  all  Los 
Angeles  area  public  transit  systems. 

Summing  up  the  overall  impacts  of  the  CCIS,  the  Chief 
stated  that  implementation  of  the  pilot  system  enabled  the 
Telephone  Information  Section  to  provide  better  service  to 
more  people. 

3. 9. 1.2  Senior  Supervisor  - The  Senior  Supervisor  of  the 
Telephone  Information  Section  stated  that  implementation  of 
the  CCIS  initially  made  the  job  much  more  complicated,  as  it 
involved  coordinating  a system  which  was  unfamiliar  to  the 
agents  and  the  supervisors.  After  the  CCIS  Program  Manager 
was  appointed,  the  job  became  much  easier.  The  Program 
Manager's  most  important  contribution  was  in  providing  tech- 
nical management  for  CCIS  operations  while  effectively  dealing 
with  the  agents'  problems.  In  effect,  the  Program  Manager 
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took  the  load  of  direct  CCIS  management  off  the  Senior  Super- 
visor's back.  The  Program  Manager,  incidentally,  was  supported 
by  a full-time  assistant  who  performed  the  data  base  updating 
function.  The  'job  of  Senior  Supervisor  is  presently  80% 
paperwork,  comprised  of  keeping  employee  time  records,  recording 
call  counts  from  the  ACD  counter  (15-20  minutes  per  hour)  and 
completing  route  activity  reports.  Much  of  this  work  could 
be  done  automatically,  with  minor  additions  to  CCIS  reporting 
functions.  If  such  software  changes  are  made,  the  CCIS  could 
decrease  the  Senior  Supervisor's  overall  paperwork  load  by  at 
least  50%  and  permit  the  performance  of  more  dedicated  super- 
vision work  with  subordinate  supervisors  and  agents.  The  Senior 
Supervisor  presently  has  a staff  of  15  regular  supervisors,  6 
of  which  are  on  duty  at  any  given  time. 

Regarding  the  impact  of  CCIS  on  the  Supervisor's  job,  the 
Senior  Supervisor  agreed  with  the  Information  Section  Chief's 
opinion  that  the  regular  agents  currently  have  the  edge  over 
the  supervisors  in  terms  of  information  knowledge  and  pro- 
cessing skill.  By  removing  the  updating  function  from  the’ 
supervisor's  job,  the  CCIS  could  change  this  situation  by 
giving  the  supervisor  position  greater  attractiveness  and 
responsibility;  supervisors  would  possess  more  system  knowledge 
by  virtue  of  greater  training  and  experience.  Supervisors 
would  also  have  the  time  to  troubleshoot  system  problems 
occurring  on  the  floor.  Finally,  agents  would  have  more 
professional  incentive  to  compete  for  the  supervisor  position 
(at  present  there  is  only  a $2  per  hour  difference  in  salary 
between  supervisor  and  agent) . 

3. 9. 1.3  Senior  Instructor  - The  Senior  Instructor  has 
responsibility  for  directing  the  initial  training  of  all 
newly-hired  agents.  Traditionally,  the  Senior  Instructor  and 
two  other  full-time  instructors  conducted  training  courses  of 
8 to  9 weeks'  duration,  during  which  agents  began  to  practice 
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on  telephone  equipment  in  the  sixth  week.  By  reducing  the 
memorization  requirements  of  this  manual  course,  the  Senior 
Instructor  estimated  that  the  CCIS  should  be  able  to  decrease 
initial  agent  training  time  from  the  present  8 to  9 weeks 
down  to  4 to  6 weeks.  It  still  will  be  necessary  to  teach 
trainees  the  basics  of: 

• Service  area  geography 

• Call  management  strategies  and  communications 

• Basic  SCRTD  data  on  routes,  schedules,  and  fares 
The  new  training  schedule  might  involve  simulated  calls  after 
two  weeks  and  "live"  calls,  under  the  supervision  of  an 
instructor,  after  three  to  four  weeks.  Each  instructor  would 
work  with  two  agents  at  any  given  time. 

Regarding  the  job  of  instructor,  the  Senior  Instructor 
mentioned  that  approximately  25%  of  instructor  time  is  pre- 
sently dedicated  to  updating  the  34  sets  of  Summary  Books  in 
the  Instruction  Office.  An  additional  five  to  six  hours  per 
week  of  instructor  time  is  spent  proofing  the  printed  material 
for  accuracy.  The  CCIS  could  decrease  or  eliminate  these 
efforts  with  some  form  of  centralized  updating;  however,  an 
alternate  form  of  quality  control,  with  staffing,  would  have 
to  be  established  for  the  computer  system.  The  cost  impact 
of  these  measures  is  unclear  at  present;  however,  some  insight 
into  labor  savings  can  be  gained  by  using  the  present  instructor 
base  salary  level  of  $1725  per  month. 

Regarding  the  job  of  Information  Section  Supervisor,  the 
Senior  Instructor  observed  that  supervisors  at  present  really 
only  perform  the  duties  of  clerks  in  terms  of  updating  the  95 
sets  of  Summary  Books  used  by  the  agents.  The  CCIS  could 
"free  up"  their  time  for  more  professional  management  duties 
such  as  promoting  call  count  productivity  and  monitoring  the 
public  relations  aspects  of  the  telephone  information  service. 
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On  other  personnel  related  matters,  the  Senior  Instructor 
noted  that  the  CGIS  may  make  it  easier  and  cheaper  to  recruit 
and  train  personnel.  The  problem  in  this  area  is  demonstrated 
by  the  statistics  on  the  last  SCRTD  recruitment  effort:  600 
individuals  responded  to  a newspaper  job  advertisement,  400 
took  the  test  at  Los  Angeles  City  College,  150  passed,  50 
were  interviewed,  18  reported  for  training,  3 graduated  from 
the  course  and  one  of  these  3 has  since  resigned. 

Although  this  is  the  least  productive  SCRTD  recruitment 
effort  to  date,  it  does  provide  some  insight  into  the  resources 
expended  in  this  overall  effort.  Major  reasons  for  losing 
trainees  include  the  relatively  low  agent  pay  scale  ($5.34  per 
hour) , the  difficulty  in  learning  the  job  and  the  repetitive 
nature  of  the  tasks.  The  CCIS  may  help  address  this  recruit- 
ment problem  by  opening  up  the  job  to  both  older  personnel  and 
part-time  workers. 

3.9. 1.4  CCIS  Program  Manager  - The  CCIS  Program  Manager,  who 
was  appointed  to  direct  daily  CCIS  operations  as  of  February  1, 
1981,  reports  to  the  Senior  Supervisor.  The  Program  Manager  is 
assisted  by  a Deputy  Program  Manager  whose  job  consists  almost 
entirely  of  performing  CCIS  update  functions,  and  who  is  required 
to  perform  liaison  tasks  with  other  SCRTD  departments  in  order 
to  obtain  the  most  accurate  data  base  possible  for  the  CCIS. 

With  the  cooperation  of  the  Senior  Supervisor,  one  of  the 
Program  Manager's  first  tasks  was  to  write  a new  training  manual 
for  the  agents  employing  user-oriented  language.  This  manual  was 
well  received  by  the  work  force.  Using  this  manual,  agents 
were  trained  in  a "refresher"  course  for  a full  day  in  groups 
of  five.  A typical  training  exercise  would  involve  one  agent 
using  the  CCIS  with  the  other  4 agents  observing.  Emphasis  in 
the  refresher  course  was  placed  on  explaining  why  the  system 
chose  the  answers  shown  on  the  response  screen.  The  subsequent 
certification  process  involved  a written  pass/fail  test;  a 
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failed  written  test  resulted  in  an  on-line  CCIS  operation  test 
for  certification.  A total  of  42  agents  successfully  completed 
this  refresher  course. 

The  Program  Manager  stated  that  interesting  training  insights 
were  gained  from  working  with  two  temporary  workers  (Kelly 
girls)  in  the  Information  Section;  these  individuals  were  of 
average  skill  level  and  were  trained  and  working  the  CCIS 
telephones  within  one  week.  This  experience  may  indicate  that 
training  time  for  new  agents  can  be  decreased  even  below  the 
projected  four  to  six  week  figure  stated  by  the  Senior  Instructor. 
It  should  be  emphasized,  however,  that  these  two  temporary 
workers  already  possessed  highly  developed  typing  skills,  which 
many  of  the  regular  agents  lacked. 

The  CCIS  Program  Manager  had  charge  of  ensuring  operational 
quality  of  the  system  during  the  90-day  test  period  (February 
22  - May  22) . From  an  operations  standpoint,  the  90-day  test 
period  was  broken  down  into  three  30-day  periods.  The  first 
month's  effort  focused  on  promoting  usage,  monitoring  agent 
progress,  receiving  agent  feedback  and  developing  standards 
for  performance.  The  second  month  of  the  test  involved  the 
development  of  the  "load  factor"  ranking  to  indicate  the  degree 
of  rush  hour  ridership  on  the  bus  routes  presented  on  the 
response  screen;  this  was  an  effort  to  direct  passenger  demand 
to  available  bus  system  capacity.  Agents  were  also  interviewed 
during  this  period  and  an  attitude  change  was  evident  involving 
a preference  for  the  CCIS  over  the  manual  system.  The  third 
month  focused  on  attaining  a steady-state  operation  with  high 
productivity  levels.  Agents  demonstrated  an  increasing  ability 
to  handle  relatively  simple  requests  (e.g.,  schedule  times) 
using  the  CCIS;  initially  the  system  was  largely  used  to 
process  more  difficult  queries. 

The  Project  Manager  indicated  overall  productivity  results 
in  the  third  month  were  not  as  high  as  expected;  this  result 
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was  attributed  to  a certain  amount  of  unaccounted  non-work  time 
(i.e.,  between  calls)  probably  resulting  from  checking  material, 
breaks,  etc.,  as,  discussed  in  Section  3.3.3. 

System  retrieval  times  however,  were  quite  satisfactory  and 
generally  ranged  from  four  to  eight  seconds  depending  upon  the 
time  of  day. 

The  Program  Manager  reiterated  opinions  expressed  by  others 
regarding  the  impact  of  the  CCIS  on  the  job  of  the  supervisor. 

The  Manager  also  stated  that  the  CCIS  data  base  is  far  more 
accurate  than  the  manual,  despite  the  best  efforts  of  the 
supervisors.  This  accuracy,  he  said,  can  have  a very  positive 
effect  on  the  agent's  confidence. 

The  CCIS,  the  Program  Manager  stated,  can  be  of  use  to  the 
SCRTD  Planning  department  in  the  assessment  of  the  need  for  new 
bus  routes  by  aggregating  origins  and  destinations  of  callers 
using  the  log  file. 

3.9.2  Customer  Relations 

3. 9. 2.1  Director  of  Customer  Relations  - the  Director  of  Customer 
Relations,  who  oversees  all  of  the  SCRTD ' s marketing  functions, 
including  the  Telephone  Information  Section,  stated  that  the 
pilot  CCIS  project  was  an  expression  of  the  SCRTD ' s confidence 
in  the  ability  of  computer  technology  to  help  transit  agents. 

The  Director's  stand  has  been  to  support  the  CCIS  providing  it 
can  do  the  job  (accurate,  responsive  information)  and  can  be 
accepted  by  the  user.  From  a cost  perspective,  the  Director 
stated  that  there  is  no  objective  of  decreasing  the  number  of 
agents  through  the  use  of  this  technology.  Rather,  ideal 
implementation  objectives  would  be  elimination  of  the  manual 
information  retrieval  system  and  higher  productivity  of  the 
work  force. 
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When  asked  to  comment  upon  the  relative  success  of  the  CCIS 
in  meeting  its  objectives,  the  Director  observed  that  after  an 
extended  "limbo"  period,  a 6-month  concentrated  effort  seems  to 
have  brought  the  user  and  the  system  together.  The  interviewer 
discussed  the  fact  that  the  attainment  of  a reasonable  level 
of  success  in  the  pilot  program  does  not  automatically  imply 
success  in  a full  systemwide  implementation  effort;  this 
possibility  was  acknowledged.  Interdepartmental  issues,  the 
Director  stated,  clearly  had  to  be  part  of  the  CCIS  expansion 
decision. 

3.9.3  Scheduling  Department 

3. 9. 3.1  Advance  Planner  - The  advance  Planner  in  the  SCRTD ' s 
Scheduling  Department  stated  that  his  responsibility  was  the 
establishment  of  basic  operating  schedules  for  buses  on  all 
system  routes.  This  procedure  involves  receiving  the  stop 
list  from  the  Stops  and  Zones  Department  and  keypunching  each 
stop  with  associated  geographic  data  (digitized  census  coordi- 
nates, census  tract,  county)  to  produce  an  integrated  master 
list  of  stops.  This  process  is  accomplished  on  a dedicated 
IBM  1401  computer  system.  The  master  list  is  then  used  to 
describe  bus  routes  in  terms  of  bus  stops.  The  Scheduling 
Department's  bus  stop  - bus  route  information  is  then  used  by 
the  Planning  Department  as  a data  base.  For  CCIS  data  base 
development  the  Scheduling  Department  provided  tapes  from 
their  IBM  1401  system  to  be  read  into  the  Univac  1106  mainframe. 

The  CCIS,  in  the  opinion  of  the  Advance  Planner,  has 
potential  applications  to  several  areas  related  to  the 
Scheduling  Department.  First,  an  integrated  transit  systemwide 
data  base  would  provide  a more  efficient  means  of  developing 
the  integrated  master  list  of  stops.  This  integrated  transit 
system  data  base  would  also  assist  the  SCRTD  in  determining 
overall  service  delivery  policy,  which  must  be  developed  based 
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on  population  and  ridership;  the  central  data  base  could  be 
used  to  describe  each  route  by  the  stops,  and  to  calculate 
route  miles  for  each  route.  This  automated  calculation  of 
route  miles  could  also  be  used  to  determine  the  cost  of 
contract  services  provided  by  SCRTD  outside  the  Los  Angeles 
County  area. 

In  view  of  the  Advance  Planner,  the  strategy  of  auto- 
matically updating  a systemwide  CCIS  data  base  with  schedule 
data  is  a worthwhile  one.  However,  it  depends  on  two 
operational  requirements:  first,  the  conversion  of  the  IBM 
1401  programs  for  Univac  requirements;  and  second,  the 
installation  of  a planned  minischeduler  for  use  as  an  inter- 
active scheduling  tool.  Completion  of  these  tasks  would  allow 
the  Scheduling  Department  to  consider  direct  involvement  in 
updating  a systemwide  CCIS  data  base.  When  asked  to  comment  on 
the  magnitude  of  the  cost  savings  associated  with  the  develop- 
ment of  this  integrated  systemwide  data  base,  the  Advance 
Planner  stated  that  the  full  system  CCIS  would  not  provide 
huge  savings  to  the  Scheduling  Department,  but  it  would  provide 
several  opportunities  for  doing  its  business  much  more 
efficiently. 


3.9.4  Stops  and  Zones 

3.9.4. 1 Supervisor  - The  Supervisor  of  the  Stops  and  Zones 
Department  of  the  SCRTD  is  responsible  for  the  entire  SCRTD 
system  of  bus  stops  and  zones.  This  activity  includes  mainte- 
nance of  a word  processor  listing  of  all  30,000  stops  and 
zones  in  the  SCRTD  service  area.  The  supervisor  is  also 
responsible  for  making  changes  in  stops  and  zones;  these  can 
result  from  new  routes,  abondoned  routes,  new  buildings, 
handicapped  service  curb  improvements,  etc.  The  Stops  and 
Zones  Department  provides  a list  of  stops  and  zones  to  the 
Scheduling  Department  for  their  use  in  developing  time  tables 
and  headway  sheets.  Finally,  the  Supervisor  is  responsible  for 
stop  signage  and  other  hardware  used  to  mark  stop  locations. 
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The  Supervisor  has  not  been  involved  in  the  CCIS  program 
and  was  not  aware  of  its  detailed  features.  However,  the 
Supervisor  did  note  that  the  system  may  be  valuable  in  two 
areas  relating  to  the  Stops  and  Zones  Department.  First,  an 
integrated  transit  system  data  base,  including  the  listing  of 
stops  and  zones,  would  appear  to  be  of  value  given  the 
exchange  of  transit  data  between  different  SCRTD  departments. 
Second,  CCIS  printed  output  may  be  useful  in  the  Informational 
Signing  Program.  This  program  involves  posted  sign  boxes  at 
each  stop  which  contain  route  and  schedule  information;  these 
are  serviced  frequently  due  to  route  and  schedule  changes,  as 
well  as  vandalism. 


3.9.5  Planning  Department 

3.9.4. 1 Director  and  Chief  Planner  - the  Director  and  Chief 
Planner  of  the  SCRTD  Planning  Department  are  responsible  for 
planning  bus  routings  in  order  to  be  responsive  to  passenger 
demand.  The  Planning  Department  conducts  surveys  and  ridership 
checks  to  assess  the  status  of  potential  and  existing  ridership 
demand.  Ridership  is  determined  using  both  point  checks  (at 
stops)  and  riding  checks.  Data  needed  to  evaluate  routings 
include  passenger  flows  and  the  identification  of  relatively 
small  neighborhood- si  zed  geographical  demand  areas.  Conceptual 
routes  are  developed  and  subjected  to  actual  testing.  If  a 
new  route  is  approved  by  the  SCRTD,  the  Scheduling  Department 
is  directed  to  design  an  operating  schedule,  and  the  Stops  and 
Zones  Department  is  directed  to  design  and  implement  a series 
of  stops  along  the  route. 

The  Planning  Department  has  not  been  involved  to  any 
significant  degree  in  the  CCIS  pilot  project  other  than  in 
providing  route  descriptions.  In  addition,  not  much  thought 
has  been  given  to  the  potential  of  the  CCIS  to  the  Planning 
Department  on  a full  system  basis.  The  interviewer  explained 
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the  potential  of  the  CCIS  log  file  in  aggregating  call  origins 
and  destinations  by  some  level  of  geographic  detail.  The 
Director  of  Bus  Planning  acknowledged  that  such  a system 
feature  might  be^of  considerable  value  for  planning  purposes. 
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4.0  CONCLUSIONS 


A review  of  the  quantitative  and  qualitative  data  collected 
by  the  system  evaluation  contractor  and  presented  in  the  pre- 
ceeding  chapters  leads  to  generally  positive  findings  concerning 
the  performance  of  the  CCIS  in  its  pilot  demonstration.  While 
the  implementation  process  was  rather  extended  and  somewhat 
laborious  due  to  a variety  of  internal  and  external  problems, 
trends  in  the  data  collected  show  that  most  of  these  problems 
have  been  corrected  and  the  system  is,  for  the  most  part, 
currently  functioning  as  originally  intended.  The  SCRTD  manage- 
ment, the  system  design  contractor,  and  the  information  agents 
using  the  system  all  demonstrated  a great  deal  of  dedication  and 
perseverance  in  bringing  about  this  result. 

This  generally  favorable  assessment  of  system  performance 
can  be  supported  by  a variety  of  data  which  indicate  that  the 
CCIS  is  achieving  a number  of  its  intended  implementation  objec- 
tives. Among  these  objectives  are: 

• Increased  agent  productivity  resulting  from  faster  data 
retrieval . 

• Increased  accuracy  and  consistency  of  agent  responses. 

• A reduction  in  training  time  required  for  new  agents, 
and  in  on-the-job  experience  required  for  full  pro- 
ficiency. 

• Improved  capability  to  rapidly  incorporate  changes  in 
the  reference  data  base. 

• An  improvement  in  the  agent's  overall  job  satisfaction. 

• Expanded  capabilities  to  collect  data  on  agent  and 
system  performance. 

• Use  of  the  transit  and  geographic  data  base  supporting 
the  system  as  a data  resource  by  other  transit  authority 
departments  such  as  those  dealing  with  route  planning, 
stop  locations,  and  service  schedules. 
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A discussion  of  CCIS  performance  in  each  of  these  objective 
areas  follows. 

4.1  AGENT  PRODUCTIVITY 

Data  collected  during  the  course  of  the  evaluation  leads  to 
rather  conflicting  conclusions  concerning  system  influence  on 
agent  productivity.  Under  rigidly  controlled  conditions  during 
the  experiment  conducted  by  the  evaluation  contractor,  the 
intermediate  and  advanced  CCIS  agents  were  appreciably  faster 
in  producing  responses  to  the  test  questions  than  were  the 
manual  and  mixed  mode  agents.  Similarly,  projected  call  counts 
on  the  CCIS  log  file,  as  discussed  in  Section  3.3.3,  for  CCIS 
agents  averaged  almost  30  per  hour,  a 50%  increase  over  the 
SCRTD ' s manual  standard  of  20.  Yet  these  same  agents  during 
the  90-day  evaluation  period  did  not  perform  at  this  30-per- 
hour  figure,  in  which  time  spent  between  calls  is  assumed  to  be 
zero.  Their  overall  percentage  of  calls  answered  vs.  calls 
lost  is  quite  comparable  to  that  of  the  manual  (non-Van  Nuys 
Trunk)  agents;  both  groups  answered  an  average  of  about  60%  of 
total  incoming  calls,  although  this  result  might  be  a function 
of  manpower  allocation  and  caller  demand  patterns  and  not  of 
the  CCIS.  In  addition,  the  CCIS  agents'  self- admitted  call 
counts  on  both  the  agent  "before"  and  "after"  opinion  surveys 
averaged  approximately  20  per  hour. 

The  conclusion  to  be  drawn  from  these  conflicting  data, 
developed  through  consultation  with  the  CCIS  Program  Manager, 
is  that  the  system  can  indeed  enable  experienced  agents  to 
approach  the  rate  of  30  calls  per  hour.  However,  it  would 
appear  that  the  stress  experienced  by  an  agent  working  at  this 
pace  is  too  great  for  it  to  be  sustained  for  extended  periods. 
The  Program  Manager  stated  that  the  working  styles  of  CCIS 
agents  can  be  classified  into  two  separate  categories.  Agents 
in  the  first  category  tend  to  work  at  a relatively  steady  pace 
of  slightly  over  20  calls  per  hour,  with  brief  break,  if  any, 
between  calls.  Agents  in  the  second  category  tend  to  work  in 
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spurts,  handling  a large  number  of  calls  in  a brief  period  (at 
the  30  call  per  hour  rate)  and  then  taking  extended  breaks  to 
"cool  down".  The  end  result  of  both  styles  is  a call  rate 
approximating  20  per  hour,  as  reported  on  the  agent  surveys  and 
observed  in  practice. 

The  CCIS  Program  Manager  noted  that  Information  Section 
management  felt  the  steady-pace  working  style  of  the  first 
category  was  clearly  preferable,  as  it  tends  to  exert  much  less 
stress  on  the  agent  while  producing  the  same  level  of  performance. 

A prime  advantage  of  CCIS  over  manual  retrieval,  the  Program 
Manager  noted,  is  that  all  calls,  even  those  involving  the  most 
complex  itineraries,  are  of  relatively  uniform  duration  and 
place  more  uniform  demands  on  an  agent's  skills.  CCIS  agents 
are  thus  enabled,  much  more  so  than  their  manual  counterparts, 
to  develop  a steady  working  pace.  In  the  long  run,  the  Program 
Manager  stated,  agent  development  of  typing  and  call  management 
skill  will  produce  gradual  increases  in  this  steady  pace. 

In  the  summer  of  1981,  the  installation  of  new  telephone 
equipment  enabled  Information  Section  Management  for  the  first 
time  to  monitor  and  control  agent  break  time.  A maximum  pro- 
ductivity for  the  entire  Section  increased  to  a rate  of 
approximately  25  calls  per  agent  per  hour.  Agents  were  thus 
forced  to  adopt  the  steady-state  working  pace.  Under  these 
conditions,  the  potential  for  increased  productivity  has  been 
realized,  although  this  result  is  not  directly  attributable  to 
the  implementation  of  the  CCIS. 

4.2  RESPONSE  ACCURACY  AND  CONSISTENCY 

The  CCIS  data  base,  even  for  the  San  Fernando  Valley  demonstra- 
tion, is  large  and  complex,  and  problems  encountered  in  its 
creation  (digitizing  street  addresses,  for  example)  left  it 
filled  with  numerous  "bugs".  Similarly,  software  routines 
developed  for  itinerary  selection  initially  made  certain  over- 
optimistic  assumptions  concerning  appropriate  walking  distances. 
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transfer  times,  allowances,  etc.  Over  an  extended  period, 
refinements  and  corrections  developed  in  consultation  with 
Information  Section  management  and  in  response  to  agent 
"trouble  reports"  have  gradually  reduced  this  accuracy  problem 
to  nearly  zero. 

Agent  responses  to  the  "before"  and  "after"  surveys  con- 
cerning system  response  accuracy  show  a decided  shift  towards 
acceptance  of  CCIS  responses.  In  the  "before"  survey,  only  40% 
of  the  agents  felt  CCIS  accuracy  was  better  than  manual,  while 
60%  felt  that  it  was  about  the  same  or  worse.  In  the  "after" 
survey,  77%  felt  that  CCIS  accuracy  was  better,  while  23%  felt 
that  it  was  about  the  same  or  worse.  Results  of  the  controlled 
experiment  also  show  that  agents  using  the  CCIS  produced  responses 
that  were  graded  at  least  as  high  on  average  as  those  of  manual 
agents,  and  those  using  CCIS  with  manual  backup  (the  mixed 
mode)  produced  the  highest  quality  responses.  It  should  be 
noted  that  the  pilot  CCIS  deployment  is  actually  such  a "mixed" 
mode  configuration  since  all  CCIS  agents  have  a complete  set  of 
manual  reference  materials  at  their  work  stations. 

It  cannot  be  overemphasized  that  the  debugging  and  enhance- 
ment process  was  an  interactive  one  in  which  agents  and  Infor- 
mation Section  management  were  active  participants.  This 
interaction  not  only  resulted  in  greater  accuracy  of  the  data 
base  and  a more  workable  system,  but  also  led  to  increased 
confidence  on  the  part  of  the  agents  that  they  were  contributing 
to  the  usefulness  of  the  system.  It  is  highly  doubtful  that 
the  pilot  demonstration  would  have  been  nearly  so  successful 
without  the  establishment  of  mechanisms  to  facilitate  this 
interaction.  The  CCIS  Working  Committee,  Program  Manager, 
and  the  "trouble  report"  system  are  all  examples  of  such 
mechani sms. 
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4.3  AGENT  TRAINING 


Interviews  with  Information  Section  management  indicated 
that  perhaps  the  greatest  positive  impacts  exerted  by  the 
CGIS  on  the  Section's  operations  could  be  realized  in  the 
area  of  agent  training.  By  drastically  reducing  the  need  for 
agent  memorization  of  geographic  and  route  data,  the  standard 
agent  training  period  can  be  cut  at  least  in  half.  If  trainees 
already  possess  typing  skills,  this  training  period  may  be 
even  shorter;  the  CGIS  Program  Manager  cited  the  experience 
of  training  two  Kelly  Girl  temporary  employees  to  reasonable 
proficiency  on  the  system  in  only  one  week.  These  two 
employees,  however,  are  the  only  ones  trained  to  date  using 
only  the  GGIS. 

This  reduced  training  time  has  strong  potential  for  benefits 
in  the  areas  of  improved  personnel  recruitment  and  retention. 

The  work  force  from  which  agents  are  recruited  can  be  greatly 
expanded  to  include  older  and  part-time  employees,  neither  of 
which  was  suited  for  the  rigorous  traditional  agent  training 
course.  It  was  the  opinion  of  Information  Section  management 
that  older  employees  are  generally  more  dependable,  and  inclusion 
of  part-time  employees  in  the  agent  work  force  would  allow 
management  more  flexibility  than  it  presently  enjoys  in  the 
scheduling  of  full-time  agent  vacations  and  other  benefits. 

The  long-term  implication  of  these  training  benefits  is  signi- 
ficant cost  savings  to  the  Telephone  Information  Section, 
although  the  lack  of  data  available  makes  these  savings  rather 
difficult  to  quantify.  It  is  quite  possible  that  such  shifts 
in  personnel  management  policy  might  receive  opposition  from 
the  information  agents'  union  if  they  are  perceived  as  a job 
threat  to  the  full-time  agents. 

4.4  UPDATING  OF  DATA  BASE 

Another  potential  major  improvement  in  Information  Section 
Operations  brought  about  as  a result  of  GGIS  implementation 
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is  the  capability  to  perform  reference  data  base  updating 
functions  on  a centralized  basis.  One  full-time  employee  at 
a single  terminal  can  perform  the  data  base  updating  tasks 
presently  performed  almost  full-time  by  15  supervisors. 

While  manual  references  would  still  probably  not  be  eliminated 
entirely,  for  backup  use  in  case  of  computer  malfunctions, 
for  example,  their  number  would  be  greatly  reduced  from  the 
95  complete  sets  currently  maintained  by  the  supervisors.  As 
a result,  considerable  labor  savings  are  anticipated  from 
centralized  updating  of  a service-area-wide  CCIS  implementation. 

A side  benefit  of  centralized  data  base  updating  mentioned 
by  almost  all  of  the  Information  Section  management  in  their 
interviews  would  be  an  improvement  in  the  job  of  supervisor. 

At  present,  supervisors  spend  practically  all  their  time 
performing  update  functions.  As  a result,  their  skills  as 
agents  tend  to  gradually  diminish  to  the  point  where  they  are 
inferior  to  those  agents  with  less  job  experience.  Centralized 
updating  would  free  the  supervisors  to  perform  the  functions 
for  which  their  jobs  were  originally  created:  troubleshooting' 
and  solving  agent  problems  on  the  floor  as  they  occur. 

4.5  AGENT  JOB  SATISFACTION 

CCIS  agent  responses  to  the  "before"  and  "after"  surveys 
give  strong  support  to  the  hypothesis  that  implementation  of 
the  CCIS  has  led  to  an  increase  in  agent  job  satisfaction. 

In  the  "before"  survey,  67%  of  responding  agents  rated  their 
job  satisfaction  as  excellent  or  good,  while  33%  rated  it  as 
fair  or  poor.  In  the  "after"  survey,  75%  of  the  agents 
responding  rated  their  job  satisfaction  excellent  or  good  and 
25%  rated  it  as  fair  or  poor.  In  the  "before"  survey,  52%  of 
agents  responding  described  CCIS  influence  on  their  job 
satisfaction  as  "positive",  44%  responded  "No  influence,"  and 
only  4%  responded  "negative."  By  contrast,  in  the  "after" 
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survey,  84%  of  the  responses  to  the  same  question  indicated 
"positive"  influence,  while  only  16%  indicated  "no  influence," 
and  there  were  no  "negative"  responses.  In  addition,  75%  of 
manual  agents  responding  on  their  "after"  survey  felt  that 
learning  to  use  CCIS  would  increase  their  job  satisfaction, 
while  25%  felt  that  it  would  decrease  job  satisfaction. 

The  implication  of  these  results  is  that  a sizable  per- 
centage of  agents  felt  that  CCIS  influenced  their  job  posi- 
tively, and  that  this  percentage  showed  an  increasing  trend 
over  the  course  of  the  evaluation  period.  One  agent  summarized 
the  influence  of  CCIS  on  her  job  rather  humorously:  "I  don't 
have  to  work  anymore;  I can  just  talk  all  day." 

4.6  DATA  COLLECTION 

The  development  of  the  CCIS  log  file  as  a data  collection 
tool  during  the  course  of  the  system  evaluation  demonstrated 
clearly  the  ability  of  the  system  to  keep  accurate  records  on 
all  aspects  of  system  performance.  Far  more  data  is  now 
available  to  Information  Section  management  concerning  the 
performance  of  each  CCIS  agent  and  of  the  entire  system  than 
was  previously  available  with  the  less-reliable  ACD  data. 
Furthermore,  expanded  data  collection  could  free  the  Senior 
Supervisor  from  a great  deal  of  paperwork  involving  ACD 
counts,  time  sheets,  etc. , yielding  more  time  to  work  with 
the  supervisors  and  agents  on  the  floor. 

4.7  INTEGRATION  WITH  OTHER  DEPARTMENTS 

Although  a CCIS  Working  Committee  has  been  established 
for  the  purpose  of  inter-departmental  coordination  of  CCIS 
activities,  the  full  benefits  of  a totally  integrated  data 
base  incorporating  stops,  zones,  routes,  and  schedules  have 
yet  to  be  fully  realized  at  the  SCRTD.  This  situation  is 
the  result  of  several  factors: 
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• The  CCIS  is  currently  limited  to  the  San  Fernando 
Valley  in  its  pilot  demonstration; 

• The  Stops  and  Zones  and  Scheduling  Departments 
current*ly  use  their  own  data  processing  systems 
which  are  incompatible  with  the  Univac  1106 
supporting  the  CCIS; 

• Management  of  other  SCRTD  departments,  which 
operate  fairly  autonomously,  appear  to  have  little 
detailed  knowledge  of  the  characteristics  or 
potentials  of  the  CCIS. 

Should  the  proposal  to  expand  the  CCIS  to  cover  the 
entire  SCRTD  service  area  be  adopted  by  the  Board  of  Directors, 
considerable  attention  should  be  given  in  system  specifications 
to  the  creation  of  a fully  integrated  transit  data  base 
which  would  be  of  use  to  all  SCRTD  departments. 

In  conclusion,  it  is  fair  to  say  that  the  CCIS  pilot 
demonstration  has  achieved  most  of  its  intended  objectives, 
and  is  currently  functioning  with  the  support  and  confidence 
of  its  users.  Bringing  about  this  result  required  dedicated, 
hard  work  and  a spirit  of  cooperation  on  the  part  of  SCRTD 
Management,  the  system  designers,  and  the  information  agents. 
Without  this  work  and  cooperation,  the  system  would  never 
have  achieved  the  success  which  it  enjoys  today. 
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APPENDIX  A 


TOTAL  INCOMING  CALLS,  ALL  TRUNKS 
6 am  - 5 pm 


WEEK 

SUN 

MON 

TUES 

WED 

THUR 

FRI 

SAT 

WEEKLY 

AVG 

2/15-2/21 

4831 

7180 

7157 

6875 

6797 

6318 

6736 

6558 

2/22-2/28 

5004 

7589 

6891 

6377 

7854 

8221 

5984 

6846 

3/1-3/7 

4336 

5692 

6843 

6224 

6313 

6347 

5417 

5895 

3/8-3/14 

4633 

7114 

7130 

8527 

7886 

7468 

7032 

7113 

3/15-3/21 

6122 

9062 

8577 

8725 

7490 

7712 

6706 

7771 

3/22-3/28 

5159 

8638 

8703 

8010 

7475 

7241 

7056 

7469 

3/29-4/4 

5001 

8721 

9159 

8233 

8494 

6842 

7526 

7711 

4/5-4/11 

6191 

9382 

6938 

6814 

5956 

6102 

5572 

6708 

4/12-4/18 

5020 

6070 

6637 

6285 

6066 

5736 

5069 

5840 

4/19-4/25 

4500 

6793 

6209 

6290 

5747 

5753 

6189 

5923 

4/26-5/2 

4485 

6619 

6864 

6650 

6518 

7368 

5651 

6308 

5/3-5Z9 

4361 

6695 

8063 

6588 

6809 

6316 

5868 

6386 

5/10-5/16 

4420 

6888 

6480 

7367 

6578 

6848 

6096 

6382 

5/17-5/23 

4756 

6548 

9546 

7772 

7691 

* 

* 

7044 

DAILY  AVG 

4922 

7370 

7541 

7220 

6990 

6837 

6223 

V7 

MAX 

6191 

9382 

9159 

8725 

8494 

8221 

7526 

X 

MIN 

4336 

5692 

6209 

6224 

5747 

5736 

5069 

/\ 

OVERALL  AVERAGE:  67  23 
STANDARD  DEVIATION:  645 


* MISSING  FROM  ACD  DATA 

\f 

\ 

SOURCE:  ACD  TRUNK  GROUP  READINGS 


APPENDIX  B 


TOTAL  INCOMING  CALLS,  VAN  NUYS  TRUNK 
6 ar;:  - 5 pm 


WEEK 

SUN 

MON 

TUES 

WED 

THUR 

FRI 

SAT 

WEEKLY 

AVG 

2/15-2/21 

632 

1105 

1060 

989 

984 

998 

879 

950 

2/22-2/28 

774 

1151 

898 

1066 

973 

872 

866 

943 

3/1-3/7 

571 

1227 

1128 

988 

864 

873 

811 

909 

3/8-3/14 

744 

1006 

1— ■ 
o 

* 

856 

853 

805 

882 

3/15-3/21 

698 

1100 

1038 

980 

773 

999 

835 

918 

3/22-3/28 

560 

1090 

1089 

905 

823 

960 

869 

899 

3/29-4/4 

613 

743 

937 

856 

1037 

975 

904 

866 

4/5-4/11 

746 

1105 

1096 

1067 

990 

1088 

801 

985 

4/12-4/18 

710 

1398 

1549 

1231 

1244 

1198 

712 

1149 

4/19-4/25 

629 

1056 

1045 

1131 

813 

1102 

980 

965 

4/26-5/2 

603 

1158 

1014 

974 

1198 

1043 

848 

977 

5/3-5Z9 

668 

1184 

1084 

964 

989 

984 

943 

974 

5/10-5/16 

638 

1154 

1110 

1311 

1263 

1247 

1023 

1106 

5/17-5/23 

623 

1009 

1467 

1148 

1050 

* 

* 

1057 

DAILY  AVG 

658 

1106 

1110 

1047 

990 

1015 

867 

MAX 

974 

1398 

978 

856 

773 

853 

712 

X 

MIN 

560 

743 

1549 

1311 

1263 

1247 

1023 

OVERALL  AVERAGE:  972 

STANDARD  DEVIATION:  83 


* MISSING  FROM  DATA 

SOURCE:  ACD  TRUNK  GROUP  READINGS 
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APPENDIX  C 

CCIS  EXPERIMENT  - AGENT  RESPONSE  QUALITY  GRADES 


QUESTION 

MIXED 

CCIS 

MANUAL 

NUMBER 

NOV 

INT 

ADV 

NOV 

INT 

ADV 

NOV 

INT 

ADV 

1 

S-9 

S-5 

S-9 

S-3 

S-8 

S-8 

S-8 

S-8 

S-8 

2 

S-10 

S-4 

S-10 

S-5 

S-4 

S-5 

S-6 

S-8 

S-9 

3 

U 

S-8 

S-4 

S-5 

S-6 

S-5 

S-7 

S-9 

S-9 

4 

S-4 

S-5 

S-8 

S-3 

S-6 

S-8 

S-6 

S-8 

S-4 

5 

S-7 

S-6 

S-9 

U 

S-8 

S-6 

S-8 

S-4 

S-7 

6 

S-6 

S-9 

S-8 

S-9 

S-10 

S-10 

S-6 

S-6 

S-8 

7 

S-5 

S-8 

S-8 

U 

s-7 

S-10 

U 

S-2 

S-8 

8 

S-9 

S-6 

S-8 

S-7 

S-8 

s-8 

S-8 

S-8 

S-9 

9 

S-7 

S-9 

S-10 

S-7 

S-9 

U 

S-7 

S-7 

S-7 

10 

S-7 

S-5 

S-7 

U 

S-5 

S-7 

U 

S-7 

S-7 

11 

S-7 

S-9 

S-8 

U 

S-9 

S-9 

S-5 

S-9 

S-9 

12 

U 

S-8 

U 

U 

U 

U 

S-6 

U 

S-4 

13 

S-6 

S-4 

S-4 

U 

S-8 

S-5 

U 

u 

S-6 

14 

S-6 

S-7 

S-10 

S-8 

S-10 

S-9 

S-10 

s-6 

S-7 

15 

S-8 

S-8 

S-9 

S-6 

S-7 

S-9 

s-9 

S-7 

S-8 

16 

S-9 

U 

S-5 

u 

S-9 

S-9 

S-9 

U 

U 

17 

S-7 

S-9 

S-9 

S-8 

S-9 

S-10 

S-6 

S-5 

S-6 

18 

S-7 

S-9 

S-10 

S-9 

S-9 

S-10 

S-5 

S-6 

S-7 

19 

S-7 

S-9 

S-10 

S-9 

S-10 

S-10 

S-7 

S-5 

S-7 

20 

S-9 

S-9 

S-10 

S-9 

s-7 

S-8 

S-9 

S-5 

S-10 

2] 

S-4 

S-7 

S-8 

U 

S-7 

S-7 

S-7 

S-9 

S-7 

22 

S-10 

S-8 

S-7 

S-9 

U 

S-8 

S-10 

S-9 

S-9 

23 

S-9 

S-9 

S-10 

S-10 

S-10 

S-10 

S-10 

S-9 

S-10 

24 

S-9 

S-8 

S-6 

S-8 

S-8 

S-8 

S-8 

S-9 

S-8 

25 

S-9 

S-9 

S-10 

S-9 

S-10 

S-6 

S-7 

S-9 

S-9 

26 

S-5 

S-9 

S-8 

S-8 

S-8 

S-8 

S-4 

S-8 

S-9 

27 

U 

S-9 

S-10 

S-10 

S-9 

U 

S-9 

S-10 

S-9 

28 

S-9 

S-7 

S-10 

S-9 

S-10 

S-10 

S-4 

S-7 

U 

29 

S-9 

S-9 

S-9 

S-9 

S-9 

s-9 

S-9 

S-3 

S-8 

30 

S-7 

S-8 

S-7 

S-8 

S-8 

S-8 

S-6 

S-7 

S-6 
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APPENDIX  C CONT'D. 


QUESTION 

NUMBER 

MIXED 

CCIS 

MANUAL 

NOV 

INT 

ADV 

NOV 

INT 

ADV 

NOV 

INT 

ADV 

31 

S-3 

S-8 

S-8 

S-3 

S-9 

S-9 

S-8 

S-5 

S-5 

32 

S-10 

S-7 

S-10 

S-9 

S-10 

S-9 

S-10 

S-9 

S-10 

33 

S-9 

S-7 

S-9 

S-9 

U 

S-8 

S-7 

S-9 

S-9 

34 

S-9 

S-9 

S-8 

S-9 

S-9 

S-7 

S-7 

S-8 

S-9 

35 

S-8 

S-8 

S-8 

S-8 

S-8 

S-9 

S-8 

S-8 

S-8 

36 

S-7 

S-6 

S-8 

S-6 

S-4 

S-7 

S-5 

U 

S-7 
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APPENDIX  D 


REPORT  OF  NEW  TECHNOLOGY 
SCRTD  - CCIS  EVALUATION 


This  report  documents  a series  of  data  collection 
efforts  undertaken  by  Wilson  Hill  Associates,  Inc.  at 
the  Telephone  Information  Section  of  the  Marketing 
Department  of  the  Southern  California  Rapid  Transit 
District  during  1980  and  1981.  The  focus  of  these 
efforts  was  a socio-economic  evaluation  of  the  impacts 
of  automated  transit  information  systems  technology 
(ATIS) , in  which  state-of-the-art  computer  systems 
were  introduced  for  the  purpose  of  information  retrieval 
by  transit  telephone  operators.  The  prototype  computer 
system  at  SCRTD  is  known  as  the  Computerized  Customer 
Information  System  (CCIS) . For  the  first  time,  system 
performance  data  were  collected  and  tabulated  by  the 
computer  to  complement  qualitative  and  quantitative 
data  collected  by  evaluation  contractor  personnel. 

A 90-day  steady-state  data  collection  period  was  used 
to  assess  the  changes  wrought  by  automation  of  data 
retrieval.  Key  variables  in  this  assessment  were 
operator  call  productivity,  operator  job  satisfaction 
and  working  attitudes,  changes  in  jobstyles  and  center 
operations  caused  by  automation,  accuracy  and  consistency 
of  operator-supplied  information,  and  cost  savings 
attributable  to  automation. 
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